Tagged: open-source

linked

A CLI for Amazon Athena #

I've been enjoying Amazon Athena to analyze our application event data.

In case you haven't played with it yet, Athena allows you to query data in S3 using SQL. My only complaint so far is having to use the web interface to manage schemas and run queries. Since Amazon offers a JDBC driver for Athena, I decided to build my first JRuby app - a command line interface for Athena catalogs.

You can run queries:

❯ cat queries/count-by-port.sql
SELECT COUNT(*) AS count, elb_name
FROM sampledb.elb_logs
GROUP BY elb_name
ORDER BY count DESC
LIMIT 10;

❯ athena query queries/count-by-port.sql
COUNT  | ELB_NAME
-------|-------------
151901 | elb_demo_006
151886 | elb_demo_009
151753 | elb_demo_001
151284 | elb_demo_002
151062 | elb_demo_004
150503 | elb_demo_008
149934 | elb_demo_005
149122 | elb_demo_007
148761 | elb_demo_003

manage schemas,

❯ athena table show sampledb.elb_logs
CREATE EXTERNAL TABLE `sampledb.elb_logs`(
  `request_timestamp` string COMMENT '',
  `elb_name` string COMMENT '',
  `request_ip` string COMMENT '',
  `request_port` int COMMENT '',
  `backend_ip` string COMMENT '',
  `backend_port` int COMMENT '',
  `request_processing_time` double COMMENT '',
  `backend_processing_time` double COMMENT '',
  `client_response_time` double COMMENT '',
  `elb_response_code` string COMMENT '',
  `backend_response_code` string COMMENT '',
  `received_bytes` bigint COMMENT '',
  `sent_bytes` bigint COMMENT '',
  `request_verb` string COMMENT '',
  `url` string COMMENT '',
  `protocol` string COMMENT '',
  `user_agent` string COMMENT '',
  `ssl_cipher` string COMMENT '',
  `ssl_protocol` string COMMENT '')
ROW FORMAT SERDE
  'org.apache.hadoop.hive.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
  'input.regex'='([^ ]*) ([^ ]*) ([^ ]*):([0-9]*) ([^ ]*):([0-9]*) ([.0-9]*) ([.0-9]*) ([.0-9]*) (-|[0-9]*) (-|[0-9]*) ([-0-9]*) ([-0-9]*) \\\"([^ ]*) ([^ ]*) (- |[^ ]*)\\\" (\"[^\"]*\") ([A-Z0-9-]+) ([A-Za-z0-9.-]*)$')
STORED AS INPUTFORMAT
  'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
  's3://athena-examples-us-east-1/elb/plaintext'
TBLPROPERTIES (
  'transient_lastDdlTime'='1480278335')

list and rebuild partitions, and more.

Setup instructions and full usage are in the GitHub repository.

linked

Gotalk #

A beautifully illustrated README for Gotalk, an async communication protocol and library from Rasmus Andersson.

comic

journal

Will trade Atom invites for pull requests

GitHub's new Atom editor is out in limited beta.

I've been blessed with a finite number of invites to share with you, my Internet friends. One can be yours for opening a pull request on someone else's public GitHub project that makes a meaningful contribution. Here are some ideas:

Just mention me on the pull request to claim your invite.

journal

Reed Esau's growing list of Vim plugins for writers

Recently, Reed Esau has amassed quite a list of Vim plugins for writers.

journal

Patches speak louder than words

Think about submitting a pull request the next time you want to open an issue on a project.

journal

Flint: lint your project for sources of contributor friction

I wrote previously on the benefits of bootstrapping consistency. Predictable scripts for automating setup and running tests greatly reduce the friction for newcomers to a project.

Recently, I've jumped back into some of my older projects and immediately felt the pain of not having bootstrap or test scripts to get up and running. Wanting a way to quickly check a project for these missing scripts and (other items that help reduce contributor friction), I wrote Flint as a small Bash script. Here's what it does:

While Bash was a good fit for the initial version, I have some ideas for Flint that really demand a higher level language:

  • Open an issue in a GitHub repository. I'd like to run Flint in a project and just have it open an issue in the repo with each of the lint errors as TODO tasks in the issue body.
  • Add missing files based on templates. I want to ship some default licenses (or pull them from choosealicense.com), and just add them to the project automatically. Boilerplate README, and CONTRIBUTING guides would be a logical addition, too.

Rewriting in Go

Ruby is my primary language, and I've written plenty of CLI apps in Ruby, but I thought I'd give Go a shot for this one, primarily for the reasons Mitchell Hashimoto and Jeremy Saenz have written about.

It's still early, but I've enjoyed writing a command line app in Go. Jeremy's cli.go project has captured much of the declarative expressiveness I like about David Copeland's gli CLI framework for Ruby.

Special thanks to Owen and Jeffrey for some helpful code review.

See Flint on GitHub.

journal

Project badges: ease or sleaze?

Erik and Mislav have an interesting discussion going on a Faraday commit on the merits of those badges in GitHub repository READMEs.

I use them in a few of my projects, but lack of consistent size and Retina support have me leaning towards removing them.

What's your take? Ease or sleaze?

linked

LOST Island #

In case you missed it, technoweenie set up a new GitHub organization and moved Faraday and Sawyer over. Mislav and I have also moved our middleware project over and I'm maintaining a legacy fork at the old spot as a redirect.

It's a big island. If you've spotted any others or have your own Faraday-related initiatives, come join on us on LOST Island.

linked

dotfiles.github.com #

As I've written previously, I've learned a lot just by digging through dotfiles on GitHub. Adam Jahnke and I set up a guide to help others looking to fork, share, and mine them for tips and tricks.

We're starting small, but we aim to be a curated list of the best dotfile projects, files, and one liners, which we'll dispense at the @octodots Twitter handle.

journal

GitHub is a fish bowl

When you create a public repository on GitHub, the whole world can watch you work. As developers, we feel the need to say "But wait, it's not done yet!"

But that's the whole point of social coding isn't it? In addition to just sharing code, aren't we supposed seek out those developers more skilled than we are and learn a thing or two? After all, if we were only interested in deconstructing finished products, there would be no Food Network. The same goes with software. Watching a project come together is a chance to peek into how software is built.

I started watching Rick's Faraday project when it only was a few commits old (before it was even called Faraday if I recall correctly). Rick's approach to REST wrappers seemed intriguing and he graciously fielded my questions as I started using it in a few pet projects. I liked it so much I began spreading the word at conferences and meetups, and out of that came Faraday Middleware.

Faraday is now appearing in all sorts of Ruby projects and enjoys a growing network. Mislav is doing a great job moving the project forward. Speaking for myself, it's doubtful if I would have been involved at all had I not caught Rick working one weekend in the fish bowl.

linked

Damien Katz on Apache and CouchDB #

If it sounds like I'm saying Apache was a mistake, I'm not. Apache was a big part in the success of CouchDB, without it CouchDB would not have enjoyed the early success it did. But in my opinion it's reached a point where the consensus based approach has limited the competitiveness of the project. It's not personal, it's business.

Thriving open source projects have a dictator, benevolent or otherwise.

All tags