# Working with U.S. Census Bureau Microdata

**URL:** <https://discourse.julialang.org/t/working-with-u-s-census-bureau-microdata/31014>\
**Category:** Community\
**Tags:** juliadb\
**Created:** [November 12, 2019, 4:56pm UTC](https://discourse.julialang.org/t/working-with-u-s-census-bureau-microdata/31014 "2019-11-12T16:56:22Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![mthelm85](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mthelm85/32/224164_2.png) [@mthelm85](https://discourse.julialang.org/u/mthelm85)\
**Post date:** [November 12, 2019, 4:56pm UTC](https://discourse.julialang.org/t/working-with-u-s-census-bureau-microdata/31014/1 "2019-11-12T16:56:23Z")

</div>

Hello,

I’m just plugging a recent video I did on how to work with the American Community Survey public-use microdata files from the U.S. Census Bureau. I show how to download the data from the Census, save to tables to be consumed by JuliaDB, how to load the data and perform computations with the Distributed package, and then how to write a function to make estimates as well as how to compute the standard error and upper/lower bounds of a confidence interval for the estimate. Hopefully, this is helpful to anyone interested in working with this dataset.

[![](https://global.discourse-cdn.com/julialang/original/3X/b/4/b4a983d5d66f18cc1526b67ec07efc288fd82f47.jpeg "Julia Programming - Survey Data / Parallel Computing With Julia") ](https://www.youtube.com/watch?v=w2VHdGxcoJI&t=17s)

P.S. I really would like to have some people join me to do live coding/QA sessions. For example, if there are any package authors out there that would like to do some quick (30 min - 1 hour) videos with me, please reach out and let’s do it!

---

<div class="post-metadata">

**Author:** ![pontus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pontus/32/10438_2.png) [@pontus](https://discourse.julialang.org/u/pontus)\
**Post date:** [November 12, 2019, 9:44pm UTC](https://discourse.julialang.org/t/working-with-u-s-census-bureau-microdata/31014/2 "2019-11-12T21:44:26Z")

</div>

Nice job with this, thanks for sharing! Census data was next on list to tackle as a public dataset, so very timely and appreciated. Am trying to especially mashup and compare using multiple sources such as Centers for Medicare & Medicaid Services (CMS), Bureau of Labor Statistics (BLS), Census Data, etc.

- Didn’t see on github - is there a place to view the App.jl and Tables.jl files you demo’d?
- I downloaded the files and would be nice to run the code.  
BTW, it was great comparing your final calculated results using JuliaDB against the Census Bureau published values to as a validation of the process and code, very effective.  
Also, the mic quality and sound levels were good.

Have you pulled and analyzed any of the BLS data yet? There is the `BlsData.jl` package by @micahjsmith that works nicely. Would be happy to share the simple notebook I made. Here’s a time-series example plotting national unemployment rate:

 ![bls-plot](https://global.discourse-cdn.com/julialang/original/3X/1/7/17fb0aed209cd66171184dcf1ff9c583d2af8a7f.png)

Oh, IRS tax data to pull in and mash up as well, and Zips and FIPS are good to key them together (and do Geoplotting, another core piece of this so following that, and shape file processing in the geo group). IRS tax database variables

```julia
-------------------------------------------------------------------------------------------------------------------------------------------
              storage display value
variable name type format label variable label
-------------------------------------------------------------------------------------------------------------------------------------------
statefips str2 %9s * The State Federal Information Processing System (FIPS) code
state str2 %9s * The State associated with the ZIP code
zipcode str5 %9s * 5-digit Zip code
...
n03220 long %8.0g * Number of returns with educator expenses 1040:23 / 1040A:16
a03220 long %8.0g * Educator expenses amount 1040:23 / 1040A:16
n03300 long %8.0g * Number of returns with self-employment retirement plans 1040:28
a03300 long %12.0g * Self-employment retirement plans amount 1040:28
n03270 long %8.0g * Number of returns with self-employment health insurance deduction 1040:29
a03270 long %12.0g * Self-employment health insurance deduction amount 1040:29
n03150 long %8.0g * Number of returns with IRA payments 1040:32 / 1040A:17
a03150 long %12.0g * IRA payments amount 1040:32 / 1040A:17
n03210 long %12.0g * Number of returns with student loan interest deduction 1040:33 / 1040A:18
a03210 long %12.0g * Student loan interest deduction amount 1040:33 / 1040A:18
n03230 long %8.0g * Number of returns with tuition and fees deduction 1040:34 / 1040A:19
a03230 long %8.0g * Tuition and fees deduction amount 1040:34 / 1040A:19
...
n85530 long %12.0g * Number of returns with additional Medicare tax 1040:62a
a85530 long %12.0g * Additional Medicare tax amount 1040:62a
n85300 long %12.0g * Number of returns with net investment income tax 1040:62b
a85300 long %12.0g * Net investment income tax amount 1040:62b
n11901 long %12.0g * Number of returns with tax due at time of filing 1040:78 / 1040A:50 / 1040EZ:14
a11901 long %12.0g * Tax due at time of filing amount [10] 1040:78 / 1040A:50 / 1040EZ:14
n11902 long %12.0g * Number of returns with overpayments refunded 1040:75 / 1040A:47 / 1040EZ:13a
a11902 long %12.0g * Overpayments refunded amount [11] 1040:75 / 1040A:47 / 1040EZ:13a
year float %9.0g * Year
                                            * indicated variables have notes
-------------------------------------------------------------------------------------------------------------------------------------------
Sorted by: zipcode statefips agi_stub year
agi_stub:
           1 $1 under $25,000 
           2 $25,000 under $50,000 
           3 $50,000 under $75,000 
           4 $75,000 under $100,000 
           5 $100,000 under $200,000 
           6 $200,000 or more 

```

---

<div class="post-metadata">

**Author:** ![randyzwitch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/randyzwitch/32/15955_2.png) [@randyzwitch](https://discourse.julialang.org/u/randyzwitch)\
**Post date:** [November 12, 2019, 9:45pm UTC](https://discourse.julialang.org/t/working-with-u-s-census-bureau-microdata/31014/3 "2019-11-12T21:45:11Z")

</div>

Wow, I was just doing a similar thing (pulling the entire ACS 2017 5-year), except not with JuliaDB! You know the community is starting to grow when this happens 🙂

---

<div class="post-metadata">

**Author:** ![pontus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pontus/32/10438_2.png) [@pontus](https://discourse.julialang.org/u/pontus)\
**Post date:** [November 12, 2019, 9:45pm UTC](https://discourse.julialang.org/t/working-with-u-s-census-bureau-microdata/31014/4 "2019-11-12T21:45:13Z")

</div>

Oh darn, @randyzwitch is replying, should have waited for my post, this will be a good one…

---

<div class="post-metadata">

**Author:** ![randyzwitch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/randyzwitch/32/15955_2.png) [@randyzwitch](https://discourse.julialang.org/u/randyzwitch)\
**Post date:** [November 12, 2019, 9:47pm UTC](https://discourse.julialang.org/t/working-with-u-s-census-bureau-microdata/31014/5 "2019-11-12T21:47:59Z")

</div>

Ha! If I knew I was going to get such praise, I’d have written something better.

Internally at work, there’s a huge desire to have census data, so I parsed it using Julia and OmniSci.jl. Was thinking of writing that up as a blog post, then didn’t…then @mthelm85 went and did a whole video!

---

<div class="post-metadata">

**Author:** ![pontus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pontus/32/10438_2.png) [@pontus](https://discourse.julialang.org/u/pontus)\
**Post date:** [November 12, 2019, 9:57pm UTC](https://discourse.julialang.org/t/working-with-u-s-census-bureau-microdata/31014/6 "2019-11-12T21:57:45Z")

</div>

Ha, thought I’d post before you with a fun quip but too slow on the trigger - you were 3 seconds ahead.

Have been pulling the full set of BLS data down this afternoon. wget is still running on the ftp site, not sure how long it will go but it’s over 12Gb so far. The CMS database with the Cost Report data loaded for all US Hospitals is 20Gb, and have the Skilled Nursing Facility and Home Health Agency tables to load still.  
Also, the work that @fipelle is doing with `TSAnalysis.jl` and the FRED Economic dataset is pretty awesome!

> [@randyzwitch](#):
>
> Was thinking of writing that up as a blog post

Pease do!!

---

<div class="post-metadata">

**Author:** ![fipelle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fipelle/32/4772_2.png) [@fipelle](https://discourse.julialang.org/u/fipelle)\
**Post date:** [November 12, 2019, 10:55pm UTC](https://discourse.julialang.org/t/working-with-u-s-census-bureau-microdata/31014/7 "2019-11-12T22:55:58Z")

</div>

@mthelm85: Very interesting video!

@pontus: Thank you! I am very happy that someone is taking interest in `TSAnalysis.jl`. I will try to finish up the next release pretty soon, and to show more examples. I am using `FredData.jl` to interface with `Alfred`. I am quite familiar with a range of economic databases. If there is something that is not there yet, feel free to leave some comments on GitHub and I will try to work on it.

---

<div class="post-metadata">

**Author:** ![mthelm85](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mthelm85/32/224164_2.png) [@mthelm85](https://discourse.julialang.org/u/mthelm85)\
**Post date:** [November 13, 2019, 1:44am UTC](https://discourse.julialang.org/t/working-with-u-s-census-bureau-microdata/31014/8 "2019-11-13T01:44:39Z")

</div>

@pontus I created a repo for the files here: [https://github.com/mthelm85/acs](https://github.com/mthelm85/acs). I’ve not seen the BlsData.jl package, thanks so much for sharing! Can you post a link to the IRS tax database? I just did a quick search and haven’t found it but that’s a dataset I’d be super interested in.

@randyzwitch I too use Census data at work so I’d love to hear more about your process (a blog post would be great 😉).

If I get some time in the next couple of weeks I’ll try to record another video on working with the Current Population Survey. It’s a really amazing dataset and it’s released monthly so it’s great when timeliness of the data is important. Unfortunately, I’ve found it to be a lot more difficult to work with and I’m hoping to figure out how to compute standard errors for the estimates before doing the video, but it’s not as straightforward with the CPS (when compared to the ACS). For example, I’ve had to e-mail the BLS on several occasions (CPS is a joint Census/BLS survey) to figure out how they are filtering the data in order to arrive at published estimates. In my experience, the way they are filtering the data is not intuitive based on what’s available in the data dictionary and results can vary dramatically based on which variable you use (or don’t).

Anyways, thanks to all of you for the feedback and thanks a ton for sharing!! I’m wondering if it would be worthwhile to get involved in the Julia slack channel and get a public data channel going…? I work with public (government) data all the time and I use Julia to do it and I usually feel like I’m in the Wild West. It would be great to get a community going…I’m not sure what it takes to get invited to the Julia slack channel though…

---

<div class="post-metadata">

**Author:** ![pontus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pontus/32/10438_2.png) [@pontus](https://discourse.julialang.org/u/pontus)\
**Post date:** [November 13, 2019, 2:05pm UTC](https://discourse.julialang.org/t/working-with-u-s-census-bureau-microdata/31014/9 "2019-11-13T14:05:37Z")

</div>

> [@mthelm85](#):
>
> BlsData.jl package, thanks so much for sharing! Can you post a link to the IRS tax database?

Appreciate you dropping the repo!  
I don’t have a link to the tax pieces, but have sourced some other info from [nber.org](http://nber.org). Let me take a look later this week and see what I can send for links. What environment are you using, Win or Linux?  
Also, regarding the FULL BLS dataset:

```julia-auto
FINISHED --2019-11-12 21:37:16--
Total wall clock time: 7h 14m 3s
Downloaded: 1784 files, 36G in 7h 1m 38s (1.46 MB/s)

```

---

<div class="post-metadata">

**Author:** ![alejandromerchan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alejandromerchan/32/10500_2.png) [@alejandromerchan](https://discourse.julialang.org/u/alejandromerchan)\
**Post date:** [November 14, 2019, 12:40am UTC](https://discourse.julialang.org/t/working-with-u-s-census-bureau-microdata/31014/10 "2019-11-14T00:40:15Z")

</div>

@mthelm85 Thanks for your stream. I just watched all your videos and they are pretty cool! When are you doing the livestreams?

---

<div class="post-metadata">

**Author:** ![mthelm85](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mthelm85/32/224164_2.png) [@mthelm85](https://discourse.julialang.org/u/mthelm85)\
**Post date:** [November 14, 2019, 1:13am UTC](https://discourse.julialang.org/t/working-with-u-s-census-bureau-microdata/31014/11 "2019-11-14T01:13:33Z")

</div>

Randomly right now…I thought it would be easier to get people to join me but everyone is either too shy or just doesn’t want to do it for some other reason 😝. I would eventually like to get to the point where I can schedule the streams once every two weeks and always have a guest on.

When I was building a lot of front-end stuff I used to watch a JavaScript-based live stream that airs on [Twitch.tv](http://Twitch.tv) every Monday morning (check out FunFunFunction on Twitch/YouTube). The host tends to get pretty good guests from the JavaScript community to join him and they code things live (and it doesn’t always go so well, but that’s actually the point 😉). It’s really become a great place for web developers to just hang out, chat with others in the community, network, interact with the host/guest, and learn. I think that kind of thing does a lot of good for a programming language community and I’d love to see it happen with Julia. The beautiful thing about the live pair programming format is that there’s not a lot of prep work that needs to be done and there’s not any editing to do afterward so it’s not a big investment for the host or the guest.