Java

Getting Started

<dependency>
    <groupId>io.internetdata</groupId>
    <artifactId>internetdata</artifactId>
    <version>2.7.0</version>
</dependency>

implementation 'io.internetdata:internetdata:2.7.0'

Requires Java 17 or newer. HTTP is the JDK's own java.net.http.HttpClient, so there is no third-party HTTP stack to reconcile with yours.

Usage

Every database endpoint is licensed, so you need an API key carrying the db.download scope. Create one in the console, then:

import io.internetdata.InternetData;

InternetData client = InternetData.create(System.getenv("INTERNETDATA_API_KEY"));

Build the client once and keep it. It owns a connection pool, and it is thread safe.

The database calls hang off client.database(), which is where the sibling VPNDetection library keeps the same seven.

What you can see

import io.internetdata.model.Database;

for (Database family : client.database().list()) {
    System.out.println(family.getBase() + " " + family.getStanding() + " " + family.getLicenseType());
    family.getVersions().forEach(v -> System.out.println("  " + v.getId() + " " + v.getFormats()));
}

A license covers a database family (bogon_ip), while a download names a version (bogon_ip_v1), so the IDs every other method takes come from getVersions().

getStanding() is LICENSED for a live grant, EXPIRED for one whose term has ended, and UNLICENSED for a database published but never bought, which is how you discover what else exists.

What is inside one

import io.internetdata.model.DatabaseMetadata;

DatabaseMetadata meta = client.database().metadata("bogon_ip_v1");
meta.getUpdated();     // the date this build was generated
meta.getEntries();     // rows in it
meta.getSize();        // bytes, per format
meta.getSchema();      // columns, per format
meta.getSample();      // a few real rows, per format

Poll this to decide whether today's build is worth fetching, and to budget a transfer before you start one.

Downloading

import io.internetdata.DatabaseFormat;
import java.nio.file.Path;

// Streamed straight to disk, so nothing bigger than a chunk is ever held in memory.
long written = client.database().download("bogon_ip_v1", DatabaseFormat.CSVGZ, Path.of("bogon_ip_v1.csv.gz"));

A transfer that dies half way is reported rather than left on disk: the bytes land in a neighboring .part file, what arrived is checked against the length the server promised, and only a whole file is moved into place.

The API answers a download with a redirect to a time-limited URL on object storage. If you would rather run the transfer yourself, ask for that link instead. It authorizes itself, so it carries none of your credentials and can be handed to any downloader:

String url = client.database().downloadUrl("bogon_ip_v1", DatabaseFormat.CSVGZ);

The link authorizes the start of a transfer, so one already running is not interrupted when it lapses.

There is also an in-memory form, for the small end of the catalog:

byte[] raw = client.database().downloadBytes("bogon_asn_v1", DatabaseFormat.CSVGZ);

downloadBytes holds the whole file in memory, and the catalog spans a few hundred bytes to several gigabytes, so check client.database().metadata(...).getSize() before reaching for it and use download for anything you have not measured.

Each of the three downloads the build that is live unless you name one. A database.updated webhook names the build it announces and carries its sha256, so a receiver that downloads after the next publish asks for that build to get the file the hash is of:

client.database().download("vpn_ip_v1", DatabaseFormat.MMDB, Path.of("vpn_ip_v1.mmdb"), build);

A build is kept 7 days. Past that the call throws BAD_REQUEST with the message BUILD_GONE, and downloading without a build fetches the live one.

Verifying a download

import io.internetdata.model.DatabaseChecksums;

DatabaseChecksums sums = client.database().checksums("bogon_ip_v1", DatabaseFormat.CSVGZ);
sums.getSha256();

Download history

Your organization's recent attempts, newest first, refusals included, which is what answers "it stopped working":

client.database().downloads().forEach(d ->
        System.out.println(d.getCreated() + " " + d.getDatasetId() + " " + d.getOutcome()));

client.database().downloads(10);   // newest ten

Errors

Failures throw an InternetDataException carrying a kind() and a retryable() flag. It is unchecked, so it travels through a stream or a future without being wrapped first:

import io.internetdata.InternetDataException;

try {
    client.database().download("bogon_ip_v1", DatabaseFormat.MMDB, Path.of("bogon_ip_v1.mmdb"));
} catch (InternetDataException e) {
    System.err.println(e.kind() + " " + e.getMessage() + " retryable=" + e.retryable());
}

kind() is one of BAD_REQUEST, UNAUTHORIZED, FORBIDDEN, RATE_LIMITED, QUOTA_EXCEEDED, SERVER_ERROR or NETWORK. getMessage() is the API's own result code where it sent one, so a refusal reads NOT_LICENSED or LICENSE_EXPIRED rather than "403".

RATE_LIMITED is a 429 that carries Retry-After, and the library retries it for you, as it does 5xx and transport failures. QUOTA_EXCEEDED is a 429 without one, which it never retries, and it retries no other client error either. InternetData's database API has no quota, so you won't see QUOTA_EXCEEDED from it.

Sign in with OAuth (device flow)

A program running on the person's own machine can let them sign in with a browser and pick one of their API keys, instead of asking them to paste it:

InternetData client = InternetData.create();

DeviceAuthorization device = client.oauth().deviceAuthorization("your-client-id",
        new DeviceAuthorizationOptions().scope("account.read apikeys.read apikeys.reveal"));
System.out.println("Open " + device.getVerificationUri() + " and enter " + device.getUserCode());

TokenResponse token = client.oauth().pollDeviceToken("your-client-id", device);
if (token.getApikey() == null) {
    throw new IllegalStateException("no API key came back: none was picked, or it cannot be shown again");
}
InternetData keyed = InternetData.builder().apiKey(token.getApikey()).build();

A denied sign-in throws OauthAccessDeniedException and a code that ran out OauthExpiredTokenException, and client IDs are issued on request from support@internetdata.io. client.oauth().revoke("your-client-id", token.getRefreshToken()) signs the machine out again.

Sign in with OAuth (authorization code)

An app that can take a browser redirect signs the person in there instead, with a PKCE pair made for that one sign-in:

InternetData client = InternetData.create();
String redirectUri = "http://127.0.0.1:8765/callback";
Pkce pkce = client.oauth().createPkce();

String url = client.oauth().authorizationUrl("your-client-id", redirectUri, pkce.challenge(),
        new AuthorizationUrlOptions().scope("apikeys.use").state("your-state"));
// Open url in the browser. Its redirect to redirectUri carries code and state.
TokenResponse token = client.oauth().exchangeAuthorizationCode("your-client-id", code, pkce.verifier(), redirectUri);

Check that state came back as you sent it before you exchange code, which works once. The client ID can also be the https URL of a client metadata document your app serves, and such an app is never handed a key, so token.getApikey() stays null.

Register a client

An app with no client ID, and no client metadata document to use as one, registers itself, then signs in with the client_id it gets back as the client ID for authorizationUrl and exchangeAuthorizationCode:

InternetData client = InternetData.create();
ClientRegistration registration = client.oauth().register(List.of("http://127.0.0.1:8765/callback"),
        new RegisterOptions().clientName("Your App"));

Each redirect URI must be https on a public host name, or http on a loopback address, which then matches any port. Nothing is kept until a person approves the client, so an app with no client ID saved can simply register again.