social.lfx.dev is one of the many independent Mastodon servers you can use to participate in the fediverse.
The Linux Foundation is the world’s leading home for collaboration on open source software, hardware, standards, and data.

Administered by:

Server stats:

6
active users

#pyspark

0 posts0 participants0 posts today
Kedro<p>A QuantumBlack team helped bring PySpark SQL support to pandera 👏🏼 we are so proud of this open source contribution and hope to keep them coming!</p><p><a href="https://www.kdnuggets.com/2023/08/data-validation-pyspark-applications-pandera.html" target="_blank" rel="nofollow noopener noreferrer" translate="no"><span class="invisible">https://www.</span><span class="ellipsis">kdnuggets.com/2023/08/data-val</span><span class="invisible">idation-pyspark-applications-pandera.html</span></a></p><p><a href="https://social.lfx.dev/tags/python" class="mention hashtag" rel="tag">#<span>python</span></a> <a href="https://social.lfx.dev/tags/pydata" class="mention hashtag" rel="tag">#<span>pydata</span></a> <a href="https://social.lfx.dev/tags/pandera" class="mention hashtag" rel="tag">#<span>pandera</span></a> <a href="https://social.lfx.dev/tags/pyspark" class="mention hashtag" rel="tag">#<span>pyspark</span></a></p>
Kedro<p>New blog post: How to integrate Kedro and Databricks Connect 🔶</p><p>In this blog post, our colleague Diego Lira explains how to use Databricks Connect with Kedro for a development experience that works completely inside an IDE.</p><p><a href="https://kedro.org/blog/how-to-integrate-kedro-and-databricks-connect" target="_blank" rel="nofollow noopener noreferrer" translate="no"><span class="invisible">https://</span><span class="ellipsis">kedro.org/blog/how-to-integrat</span><span class="invisible">e-kedro-and-databricks-connect</span></a></p><p>Install it with</p><p>```<br />pip install databricks-connect<br />```</p><p><a href="https://social.lfx.dev/tags/kedro" class="mention hashtag" rel="tag">#<span>kedro</span></a> <a href="https://social.lfx.dev/tags/python" class="mention hashtag" rel="tag">#<span>python</span></a> <a href="https://social.lfx.dev/tags/pydata" class="mention hashtag" rel="tag">#<span>pydata</span></a> <a href="https://social.lfx.dev/tags/datascience" class="mention hashtag" rel="tag">#<span>datascience</span></a> <a href="https://social.lfx.dev/tags/databricks" class="mention hashtag" rel="tag">#<span>databricks</span></a> <a href="https://social.lfx.dev/tags/dbx" class="mention hashtag" rel="tag">#<span>dbx</span></a> <a href="https://social.lfx.dev/tags/spark" class="mention hashtag" rel="tag">#<span>spark</span></a> <a href="https://social.lfx.dev/tags/pyspark" class="mention hashtag" rel="tag">#<span>pyspark</span></a></p>
Kedro<p>kedro-datasets 1.4.0 is out! 🔶 With a new SparkStreamingDataSet!</p><p>kedro-datasets is a separate PyPI package where Kedro datasets live. ⚠️ Notice that `kedro.extras.datasets` is deprecated and will be removed in Kedro 0.19, so install the new package now!</p><p>```<br />pip install &quot;kedro-datasets==1.4.0&quot;<br />```</p><p><a href="https://social.lfx.dev/tags/kedro" class="mention hashtag" rel="tag">#<span>kedro</span></a> <a href="https://social.lfx.dev/tags/datascience" class="mention hashtag" rel="tag">#<span>datascience</span></a> <a href="https://social.lfx.dev/tags/python" class="mention hashtag" rel="tag">#<span>python</span></a> <a href="https://social.lfx.dev/tags/pydata" class="mention hashtag" rel="tag">#<span>pydata</span></a> <a href="https://social.lfx.dev/tags/spark" class="mention hashtag" rel="tag">#<span>spark</span></a> <a href="https://social.lfx.dev/tags/pyspark" class="mention hashtag" rel="tag">#<span>pyspark</span></a></p>
Henry<p>Now I’ve been here a few weeks, found the fire exits, amenities and snacks, I should probably add an <a href="https://social.lol/tags/introduction" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>introduction</span></a> post.</p><p>Hello 👋 I’m Henry, and professionally I do <a href="https://social.lol/tags/DataScience" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>DataScience</span></a> and <a href="https://social.lol/tags/DataEngineering" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>DataEngineering</span></a> in the <a href="https://social.lol/tags/Aviation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>Aviation</span></a> industry, mostly with <a href="https://social.lol/tags/python" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>python</span></a> and <a href="https://social.lol/tags/pyspark" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>pyspark</span></a></p><p>Unprofessionally I spend my time <a href="https://social.lol/tags/parenting" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>parenting</span></a> two children along with my wife and occasionally manage to find time to play <a href="https://social.lol/tags/guitar" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>guitar</span></a> 🎸 improve my use of <a href="https://social.lol/tags/emacs" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>emacs</span></a> and hack at too many half forgotten <a href="https://social.lol/tags/maker" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>maker</span></a> projects.</p>