Recent searches

No recent searches

Search options

Only available when logged in.
shakedown.social is one of the many independent Mastodon servers you can use to participate in the fediverse.
A community for live music fans with roots in the jam scene. Shakedown Social is run by a team of volunteers (led by @clifff and @sethadam1) and funded by donations.

Administered by:

Server stats:

255
active users

shakedown.social: AboutStatusProfiles directoryPrivacy policy

Mastodon: AboutGet the appKeyboard shortcutsView source codev4.4.3

#duckdb

1 post1 participant0 posts today
Olivier Leroy<p>My talks at <span class="h-card" translate="no"><a href="https://mastodon.social/@useR_conf" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>useR_conf</span></a></span> is here <a href="https://defuneste.codeberg.page/useR_2025/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">defuneste.codeberg.page/useR_2</span><span class="invisible">025/</span></a></p><p>tldr: I think storing "big" data as a parquet files, stored in s3 accessed with duckDB and wrapped in an R package is a nice way to save some of your sanity. </p><p>Now that we know that DuckDB is great let start showing how R can make it in production! 😉 </p><p>Side notes: loved using {litedown} and codeberg for the prez. Mermai.js you are also great but I am not rdy!</p><p><a href="https://fosstodon.org/tags/Rstats" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Rstats</span></a> <a href="https://fosstodon.org/tags/DuckDB" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DuckDB</span></a></p>
Ryan Kennedy<p>I’ve always known that the <a href="https://hachyderm.io/tags/DuckDB" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DuckDB</span></a> appender interface was the way to go for bulk loading data. But today I had reason to write a <a href="https://hachyderm.io/tags/Golang" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Golang</span></a> benchmark to see just how much faster it is and discovered it’s at least 250x faster (on my laptop) at inserting a bigint into a table.</p><p>I tested both in-memory and on-disk as well as testing INSERT with auto-commit and with batched commits at various batch sizes.</p><p><a href="https://gist.github.com/rkennedy-argus/9e9b2a9fe79d7b098ff40bfb4ffc0384" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">gist.github.com/rkennedy-argus</span><span class="invisible">/9e9b2a9fe79d7b098ff40bfb4ffc0384</span></a></p><p>I suppose I should test INSERTs with prepared statements, too. But I doubt they’ll put much of a dent in that difference.</p>
Ryan Kennedy<p><span class="h-card" translate="no"><a href="https://mastodon.nl/@christoffel66" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>christoffel66</span></a></span> exactly…the “highest” available in the list provided. So far the ORDER BY with list_position seems to be the clearest winner in terms of readability and not repeating itself. </p><p>This is using <a href="https://hachyderm.io/tags/DuckDB" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DuckDB</span></a>.</p>
boB Rudis 🇺🇦 🇬🇱 🇨🇦<p>I track <span class="h-card" translate="no"><a href="https://infosec.exchange/@stratosphere" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>stratosphere</span></a></span>'s posts &amp; their bot has a daily top 10 sketch IPs list. My 🧠 kept 👀 lots of "*.100" IPs &amp; I was curious how frequently they showed up.</p><p>Went back 200 posts w/GH:McKael/madonctl using both R and DuckDB.</p><p>Def block these.</p><p>— <a href="https://mastodon.social/tags/DuckDB" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DuckDB</span></a>: <a href="https://ray.so/SdMcBZa" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">ray.so/SdMcBZa</span><span class="invisible"></span></a><br>— <a href="https://mastodon.social/tags/RStats" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RStats</span></a>: <a href="https://ray.so/naTBBMS" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">ray.so/naTBBMS</span><span class="invisible"></span></a></p>
Data Quine<p>Discovering DuckDB Use Cases via GitHub - Petrica Leuca</p><p>"TL;DR: In this post, we use the GitHub API to find repositories that mention DuckDB, then use DuckDB itself to parse and query the results efficiently with SQL."</p><p>Think a lot of projects/organisations would like some of the techniques shown in this post for finding out who else is using their code on GitHub and for what purpose.</p><p><a href="https://duckdb.org/2025/06/27/discovering-w-github" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">duckdb.org/2025/06/27/discover</span><span class="invisible">ing-w-github</span></a></p><p><a href="https://datasci.social/tags/OpenSource" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>OpenSource</span></a> <a href="https://datasci.social/tags/GitHub" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>GitHub</span></a> <a href="https://datasci.social/tags/DuckDB" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DuckDB</span></a></p>
aerique<p>Great talk by Hannes Mühleisen of <a href="https://genart.social/tags/DuckDB" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DuckDB</span></a> about tables being a fundamental technology to civilization and not dismissing databases, SQL &amp; ACID just because some implementation are getting old in the tooth.</p><p>DuckDB sounds awesome and I know <span class="h-card" translate="no"><a href="https://mastodon.nl/@bert_hubert" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>bert_hubert</span></a></span> is a big fan.</p><p><a href="https://genart.social/tags/JoyOfCoding" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>JoyOfCoding</span></a> <a href="https://genart.social/tags/JoyOfCoding2025" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>JoyOfCoding2025</span></a></p>
hrbrmstr's Daily Drop<p><strong>Drop #669 (2025-06-23): Monday Morning (Barely) Grab&nbsp;Bag</strong></p><p><em>Rube Goldberg X-traction Pipeline; fplot; Color Everything in CSS</em></p><p>Something for (hopefully) everyone as we start off this brutally hot (in many parts of the northern hemisphere) terminal week of June.</p><p><strong>Stay safe out there.</strong></p> <p>Type your email…</p><p>Subscribe</p> <p><strong>TL;DR</strong></p><p><em>(This is an LLM/GPT-generated summary of today’s Drop using Ollama + Qwen 3 and a custom prompt.)</em></p><ul><li>A Rube Goldberg-inspired data pipeline is created to archive X posts into a DuckDB database, using XCancel, Inoreader, and a DuckDB script for automation (<a href="https://en.wikipedia.org/wiki/Rube_Goldberg" rel="nofollow noopener" target="_blank">https://en.wikipedia.org/wiki/Rube_Goldberg</a>)</li><li>The&nbsp;<code>{fplot}</code>&nbsp;R package automates the creation of distribution plots by detecting data types and selecting appropriate visualizations, with options for global relabeling of variables (<a href="https://lrberge.github.io/fplot/" rel="nofollow noopener" target="_blank">https://lrberge.github.io/fplot/</a>)</li><li>The CSS-Tricks article “Color Everything in CSS” provides an in-depth look at color spaces, models, and gamuts in modern web development, offering a comprehensive guide to advanced CSS color techniques (<a href="https://css-tricks.com/color-everything-in-css/" rel="nofollow noopener" target="_blank">https://css-tricks.com/color-everything-in-css/</a>)</li></ul> <p><strong>Rube Goldberg X-traction Pipeline</strong></p><p>I don’t see many mentions of&nbsp;<a href="https://en.wikipedia.org/wiki/Rube_Goldberg" rel="nofollow noopener" target="_blank">Rube Goldberg</a>&nbsp;in pop-culture settings anymore, which is a shame, since I used to enjoy poring over them in my younger days. Perhaps the reason for the lack of mentions is that many data pipelines have much in common with those complex, over-“engineerd” contraptions.</p><p>Case in point for a recent “need” of mine: I wanted a way to store posts from users on X into a DuckDB database, for archival and research purposes. I already use&nbsp;<a href="https://xcancel.com" rel="nofollow noopener" target="_blank">XCancel</a>’s ability to generate an RSS feed for an account/search, which I yank into Inoreader for the archival part (the section header shows the XCancel-generated RSS feed for the White House’s other, even more MAGA, propaganda account).</p><p>Inoreader’s API is…not great. It can most certainly be machinated (I have an R package with the function I need in it), but I really wanted a solution that let me just use DuckDB for all the work.</p><p>Then, I rememberd, if you put feeds in Inoreader folders, you can turn that folder into a JSON feed that gets updates every ~30 minutes or so. This one:</p><p>is for a series of feeds related to what’s going on in the Middle East right now.</p><p>With that JSON URL in hand, it’s as basic as:</p> <pre>#!/usr/bin/env bash# for cache bustingepoch=$(date +%s)duckdb articles.ddb &lt;&lt;EOQLOAD json;INSTALL shellfs FROM community;LOAD shellfs;CREATE TABLE IF NOT EXISTS broadcast_feed_items ( url VARCHAR PRIMARY KEY, title VARCHAR, content_html VARCHAR, date_published VARCHAR, tags VARCHAR[], authors JSON);-- this is where the update magic happensINSERT OR IGNORE INTO broadcast_feed_itemsFROM read_json('curl -s https://www.inoreader.com/stream/user/##########/tag/broadcast/view/json?since=${epoch} | jq .items[] |')SELECT url, title, content_html, date_published, tags, authors;-- Thinned out JSON content for viewing appCOPY ( FROM broadcast_feed_items SELECT content_html, -- "title" is useless for the most part since this is an X post date_published AS "timestamp", regexp_replace(authors.name, '"', '', 'g') AS handle) TO 'posts.json' (FORMAT JSON, ARRAY );EOQ</pre> <p>There are other ways to unnest the data than using&nbsp;<code>jq</code>&nbsp;and the&nbsp;<code>shellfs</code>&nbsp;DuckDB extension, but the more RG the better (for this post)!</p><p>So the final path is:</p><p>X -&gt; XCancel -&gt; XCancel RSS -&gt; Inoreader -&gt; Inoreader JSON -&gt; jq -&gt; DuckDB</p><p>with virtually no code (save for the snippet, above).</p><p>I’ve got this running as a systemd timer/service running every 30 minutes.</p><p>Later this week (when I’m done hand-coding it—yes, sans-Claude), I’ll have a Lit-based vanilla HTML/CS/JS viewer app in one of the Drops.</p> <p><strong>fplot</strong></p><p><em>(This is an <a rel="nofollow noopener" class="hashtag u-tag u-category" href="https://dailydrop.hrbrmstr.dev/tag/rstats/" target="_blank">#RStats</a> section, so def move along if that is not your cuppa.)</em></p><p>My daily git-stalking led me to finding this gem of an R package.</p><p><a href="https://lrberge.github.io/fplot/" rel="nofollow noopener" target="_blank"><code>{fplot}</code></a>&nbsp;(<a href="https://github.com/lrberge/fplot?tab=readme-ov-file" rel="nofollow noopener" target="_blank">GH</a>) is designed to automate and simplify the visualization of data distributions (something I have to do every. single. day.). Its core mission is to let folks quickly generate meaningful and aesthetically pleasing distribution plots, regardless of the underlying data type (it supports continuous, categorical, or skewed), by making spiffy choices about the appropriate graphical representation for each variable.</p><p>Functions in the package detect the nature of your data (e.g., categorical vs.&nbsp;continuous, skewed or not) and automatically selects the most suitable plot type. For example, it will not use the same visualization for a categorical variable as it would for a continuous one, and it adapts further if the data is heavily skewed.</p><p>Ergonomics are pretty dope, since you only need a single line of code to generate a plot, with the package handling the details of layout and type selection. This is particularly useful for exploratory data analysis or for folks who want quick, visually appealing graphics without extensive customization.</p><p>Tools are provided to globally relabel variable names for all plots. This is managed via the&nbsp;<code>setFplot_dict()</code>&nbsp;function, which lets us map cryptic/gosh awful or technical variable names to more readable labels that will appear in all subsequent plots.</p><p>Example usage:</p> <pre>setFplot_dict(c( Origin = "Exporting Country", Destination = "Importing Country", Euros = "Exports Value in €", jnl_top_25p = "Pub. in Top 25% journal", jnl_top_5p = "Publications in Top 5% journal", journal = "Journal", institution = "U.S. Institution", Petal.Length = "Petal Length"))</pre> <p>The typical workflow with fplot is straightforward:</p><ol><li>Load your data.</li><li>Optionally set global variable labels using&nbsp;<code>setFplot_dict()</code>.</li><li>Call the&nbsp;<code>fplot</code>&nbsp;function on your variable(s) of interest.</li><li>The package automatically determines the best plot type and layout for your data.</li></ol><p>The same function call can yield different types of plots depending on the data provided, streamlining the process of distributional analysis and visualization.</p><p>A gallery of examples and a more detailed walk-through are available on the package’s website.</p> <p><strong>Color Everything in CSS</strong></p><p>The CSS-Tricks article “<a href="https://css-tricks.com/color-everything-in-css/" rel="nofollow noopener" target="_blank">Color Everything in CSS</a>” offers a comprehensive, up-to-date exploration of how color works in CSS, moving beyond just the basics of color and background-color to cover the deeper technical landscape of color on the web. The article introduces essential concepts like color spaces, color models, and color gamuts, which are foundational for understanding how colors are represented, manipulated, and rendered in browsers today.</p><p>We’ve covered many of these individual topics before, but this is a well-crafted, all-in-one that does such a good job, I do not wish to steal any thunder from it. Head on over for to level up your CSS skills.</p> <p><strong>FIN</strong></p><p>Remember, you can follow and interact with the full text of The Daily Drop’s free posts on:</p><ul><li>🐘 Mastodon via&nbsp;<code>@dailydrop.hrbrmstr.dev@dailydrop.hrbrmstr.dev</code></li><li>🦋 Bluesky via&nbsp;<code>https://bsky.app/profile/dailydrop.hrbrmstr.dev.web.brid.gy</code></li></ul><p>☮️</p><p><a rel="nofollow noopener" class="hashtag u-tag u-category" href="https://dailydrop.hrbrmstr.dev/tag/duckdb/" target="_blank">#duckdb</a> <a rel="nofollow noopener" class="hashtag u-tag u-category" href="https://dailydrop.hrbrmstr.dev/tag/rstats/" target="_blank">#RStats</a></p>
boB Rudis 🇺🇦 🇬🇱 🇨🇦<p>Never gonna not 💙 <a href="https://mastodon.social/tags/DuckDB" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DuckDB</span></a>'s built-in bar chart function</p>
Francis 🏴‍☠️ Gulotta<p>oops <a href="https://toot.cafe/tags/til" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>til</span></a> to use <a href="https://toot.cafe/tags/duckdb" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>duckdb</span></a> to query a CSV, generate date ranges, use windowing functions to backfill data and pivot functions to make data that you can easily graph in a spreadsheet.</p><p>based upon;<br>- average solar radiation distribution over the year for my area<br>- My actual kwh production and usage for the last month (which <a href="https://toot.cafe/tags/homeassistant" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>homeassistant</span></a> gives as data change events, not hourly or daily reporting) <br>- The KWHs I've spent on AC that I expect to increase over the summer</p><p>I'm operating at 85% capacity 🙌</p>
Christos Argyropoulos MD PhD<p>Since we can't use the cloud to automate our <a href="https://mastodon.social/tags/EHR" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>EHR</span></a> analysis projects , we tried <a href="https://mastodon.social/tags/nextflow" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>nextflow</span></a> (traditionally used in bioinformatics <a href="https://www.nextflow.io/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://www.</span><span class="">nextflow.io/</span><span class="invisible"></span></a>), and it worked as a charm coordinating <a href="https://mastodon.social/tags/duckdb" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>duckdb</span></a> <a href="https://mastodon.social/tags/R" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>R</span></a> <a href="https://mastodon.social/tags/python" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>python</span></a> and within node tasks.<br>Next in line is (R)?ex <a href="https://www.rexify.org/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://www.</span><span class="">rexify.org/</span><span class="invisible"></span></a></p>
Michael Simons<p>&gt; Anyone who has worked for more than 5 minutes in an enterprise more than 30 miles outside San Fransisco know that the vast majority of information in the enterprise is cataloged and transacted via Excel spreadsheets. And if you're lucky, these spreadsheets are accessible to more than one person at a time via platforms like SharePoint.</p><p>This is large. </p><p>Or shall I say x-large? </p><p>It basically excels and will essentially solve all German Enterprise IT issues:</p><p><a href="https://github.com/gregwdata/ducklakexl" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">github.com/gregwdata/ducklakexl</span><span class="invisible"></span></a></p><p><a href="https://mastodon.social/tags/DuckDB" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DuckDB</span></a></p>
Yazad<p>A general <a href="https://techhub.social/tags/softwareengineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>softwareengineering</span></a> question, when using the word <a href="https://techhub.social/tags/embedded" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>embedded</span></a> how do you distinguish between embedded as in <a href="https://techhub.social/tags/firmware" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>firmware</span></a> vs <a href="https://techhub.social/tags/embedded" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>embedded</span></a> as in in process like a programming language like <a href="https://techhub.social/tags/lua" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>lua</span></a> that is used in a game engine or a database like <a href="https://techhub.social/tags/sqllite" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>sqllite</span></a> or <a href="https://techhub.social/tags/duckdb" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>duckdb</span></a> that (can potentially) run in the host application. As in do you use different terms like <a href="https://techhub.social/tags/embeddedsystems" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>embeddedsystems</span></a> or <a href="https://techhub.social/tags/EmbeddedSoftware" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>EmbeddedSoftware</span></a> to refer to one or the other? Or am I completely off base here?</p>
*
Esk 🐌⚡💜

howdy, #hachyderm!

over the last week or so, we've been preparing to move hachy's #DNS zones from #AWS route 53 to bunny DNS.

since this could be a pretty scary thing -- going from one geo-DNS provider to another -- we want to make sure *before* we move that records are resolving in a reasonable way across the globe.

to help us to do this, we've started a small, lightweight tool that we can deploy to a provider like bunny's magic containers to quickly get DNS resolution info from multiple geographic regions quickly. we then write this data to a backend S3 bucket, at which point we can use a tool like #duckdb to analyze the results and find records we need to tweak to improve performance. all *before* we make the change.

then, after we've flipped the switch and while DNS is propagating -- :blobfoxscared: -- we can watch in real-time as different servers begin flipping over to the new provider.

we named the tool hachyboop and it's available publicly --> github.com/hachyderm/hachyboop

please keep in mind that it's early in the booper's life, and there's a lot we can do, including cleaning up my hacky code. :blobfoxlaughsweat:

attached is an example of a quick run across 17 regions for a few minutes. the data is spread across multiple files but duckdb makes it quite easy for us to query everything like it's one table.

#sre#devops#mastodon
jmsfbs

I'm James, I've only recently started using Mastodon. Mainly after #cohost shut down.

I love #music. I play guitar and sing, solo and in a band. I live in the #bluemountains in Australia.

I'm also a software dev who's pretty active in the #JS ecosystem. I love #postgres and #duckdb I used to be more active in #fp spaces but not so much now.

Completely as a hobby I'm interested in #gamedev / #gamedesign

I'm fairly #left leaning but I don't get too political online.

#introduction
*
Marcos Huerta

I liked using #polars for my recent analysis of those large parquet files, but #duckdb did the same core computation (filtering and counting with a group by, directly from the parquet files) about 50% faster. 224ms vs 361ms. (EDITED as my earlier numbers were not the same calculations…)

But subsequent steps (normalizing the counts, creating cumulative sums) are just much more intuitive for me to write in python with polars than writing more sql.

TrendingLive feeds
About