Erlang

Getting Started

%% rebar.config
{deps, [internetdata]}.

Requires Erlang/OTP 27 or newer. There are no runtime dependencies: everything the client needs is in OTP. From Elixir, add {:internetdata, "~> 1.6"} to your mix.exs deps and call it as :internetdata.

Usage

Every database endpoint published today is authenticated, so a client needs an API key for them. Create one in the console with the db.download scope; keys are default-deny, so an existing key does not gain database access until that scope is added to it. api_key is nevertheless OPTIONAL: internetdata:new() builds a client that sends no authorization header at all, ready for a dataset served without a license.

Client = internetdata:new(#{api_key => <<"your-api-key">>}),

{ok, Databases} = internetdata:database_list(Client),
[maps:get(base, D) || D <- Databases].   % [<<"bogon_ip">>, <<"vpn_ip">>, ...]

Every call answers {ok, Term} or {error, Error}. The client is a plain term holding no process and no connection of its own, so there is nothing to close and nothing to keep alive.

The catalog

database_list/1 returns every dataset FAMILY your organization may see, with your license beside each one:

{ok, Databases} = internetdata:database_list(Client),

[begin
     io:format("~s (~s): ~s~n", [maps:get(base, D), maps:get(standing, D), maps:get(name, D)]),
     [io:format("  ~s ~p~n", [maps:get(id, V), maps:get(formats, V)])
      || V <- maps:get(versions, D)]
 end || D <- Databases].

standing is <<"licensed">> for a live grant, <<"expired">> for one whose term has ended, and <<"unlicensed">> for a database published but never bought. license_type says what your license permits you to do with the data, and is null when there is no license. internetdata:standings() and internetdata:license_types() list the values each can take, and internetdata:database_formats() the formats a download takes, for checking one that came from a flag or a config file.

A license covers a family (bogon_ip), while a download names one of its versions (bogon_ip_v1), so the ids the other calls take come from a family's versions rather than from the family itself. Old versions are frozen rather than migrated, so both stay downloadable.

What is in a database

database_metadata/2 describes one version without downloading any of it, so it is cheap to poll and it is what to check a transfer against before starting one:

{ok, Metadata} = internetdata:database_metadata(Client, <<"bogon_ip_v1">>),

maps:get(updated, Metadata).                        % <<"2026-09-04">>, the day this build was made
maps:get(entries, Metadata).                        % row count
maps:get(<<"csvgz">>, maps:get(size, Metadata)).    % bytes

schema, sample and size are keyed by format, and a sample row is keyed by the dataset's own column names. Those keys stay binaries, because they are the server's to choose; the fields the API documents are atoms. Same rule everywhere in this library.

Downloading

database_download/4 streams a database straight to a file. Nothing beyond a single chunk is ever held in memory, so the size of the dataset does not matter:

{ok, Written} = internetdata:database_download(Client, <<"bogon_ip_v1">>, csvgz, "bogon_ip.csv.gz").

The bytes go to a neighboring .part file that is renamed only once the whole body has arrived, so a transfer that dies halfway leaves neither a truncated file that reads as a complete database nor a .part for the next attempt to append to.

For a small database you can take the bytes directly. This holds the whole file in memory, and the published catalog runs from a few hundred bytes to several gigabytes, so check size first for anything you have not measured:

{ok, Bytes} = internetdata:database_download_bytes(Client, <<"bogon_ip_v1">>, csvgz).

database_download_url/3 hands back the link instead of the bytes, so you can transfer it however you like. The API answers a redirect straight to object storage and the returned URL carries no credential of yours, so it is safe to pass to another process, another machine, or curl:

{ok, Url} = internetdata:database_download_url(Client, <<"bogon_ip_v1">>, mmdb).

It authorizes the START of a transfer, so one already running is not interrupted when the link lapses.

Verifying a download

{ok, Checksums} = internetdata:database_checksums(Client, <<"bogon_ip_v1">>, csvgz),
maps:get(sha256, Checksums).

Read the checksums after the transfer rather than before it: a build published in between then shows up as a mismatch instead of passing against the digest of a file you no longer have.

Download history

database_downloads/2 lists your organization's recent download attempts, newest first, refusals included. A denial is what answers "it stopped working", and its absence answers nothing. limit defaults to 50 and the API clamps it to 200:

{ok, Attempts} = internetdata:database_downloads(Client, #{limit => 20}),

[io:format("~s ~s ~s~n", [maps:get(created, A), maps:get(dataset_id, A), maps:get(outcome, A)])
 || A <- Attempts].

Errors

A failure answers {error, Error}, where Error is a map carrying a kind and a retryable flag:

case internetdata:database_download_url(Client, <<"vpn_ip_v1">>, mmdb) of
    {ok, Url} ->
        fetch(Url);
    {error, #{kind := forbidden, message := Rc}} ->
        io:format("not licensed: ~s~n", [Rc]);
    {error, Error} ->
        io:format("~p: ~s~n", [maps:get(kind, Error), maps:get(message, Error)])
end.

kind is one of bad_request, unauthorized, forbidden, rate_limited, quota_exceeded, server_error, network or io. status carries the HTTP status where there was one, and message is the API's own result code, so NOT_LICENSED and LICENSE_EXPIRED tell you which 403 you got without having to ask us.

Note that rate_limited and quota_exceeded both arrive as HTTP 429 and are not the same thing. A rate limit is when the API faces extreme traffic bursts and so retrying later works; but a spent quota needs your allowance raised or the window to roll over. The library retries rate limits for you, but not if your quota is exceeded.

Timeouts and retries

Client = internetdata:new(#{api_key => <<"your-api-key">>, retries => 4, timeout_ms => 60000}).

timeout_ms bounds the wait for a whole request, body included, except during a download, where it bounds the wait between chunks instead: a deadline that suits a listing is the wrong one for a gigabyte, while a transfer that has stopped making progress is stalled at any size. It bounds each attempt, so a call that is retried can take longer in total, and database_downloads/2 and every oauth_* function take a timeout_ms of their own for that call alone:

{ok, Attempts} = internetdata:database_downloads(Client, #{limit => 20, timeout_ms => 5000}).

Sign in with OAuth (device flow)

A program running on a person's own machine can let them sign in with their browser and pick one of their API keys, instead of asking them to paste one.

Client = internetdata:new(),
ClientId = <<"your-client-id">>,
{ok, Device} = internetdata:oauth_device_authorization(Client, ClientId,
    #{scope => <<"account.read apikeys.read apikeys.reveal">>}),
io:format("Open ~s and enter ~s~n", [maps:get(verification_uri, Device), maps:get(user_code, Device)]),
{ok, Token} = internetdata:oauth_poll_device_token(Client, ClientId, Device),
Keyed = case Token of
    #{apikey := Key} -> internetdata:new(#{api_key => Key});
    #{} -> error(no_api_key_picked)
end.

A refusal answers {error, #{error_code := <<"access_denied">>}} and a code that expired first {error, #{error_code := <<"expired_token">>}}. Client IDs are issued on request from support@internetdata.io, and internetdata:oauth_revoke(Client, ClientId, maps:get(refresh_token, Token)) signs the machine out.