Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: initial implementation of the indexer #7

Merged
merged 31 commits into from
Sep 19, 2024
Merged
Show file tree
Hide file tree
Changes from 30 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
c53055f
feat: update our state from IPNI state
bajtos Sep 4, 2024
2ec7716
Merge branch 'main' into indexer-impl
bajtos Sep 4, 2024
fe0e9cc
fixup! finish "update our state from IPNI state"
bajtos Sep 4, 2024
9bbe3bc
start working in the advertisement walker
bajtos Sep 4, 2024
ef48905
Update indexer/lib/typings.d.ts
bajtos Sep 4, 2024
df599e1
handle advertisement with no entries
bajtos Sep 5, 2024
7482b0b
small tweaks, fix tests
bajtos Sep 10, 2024
8c612a1
feat: finish processNextAdvertisement
bajtos Sep 10, 2024
3e0d552
refactor: cleanup
bajtos Sep 10, 2024
459a14c
docs: update the design doc
bajtos Sep 10, 2024
36174d8
Merge branch 'main' into indexer-impl
bajtos Sep 10, 2024
17a05e8
walk one step
bajtos Sep 11, 2024
9a93803
refactor: move loop runners to lib
bajtos Sep 11, 2024
72da9eb
wip: walk providers independently from each other
bajtos Sep 11, 2024
39e257a
wip: don't retry failed op too often
bajtos Sep 12, 2024
a22bc7a
don't retry failed op too often, keep the IPNI state in memory only
bajtos Sep 12, 2024
c86f645
feat: detect HTTP request errors
bajtos Sep 16, 2024
a6cb5e2
Merge branch 'main' into indexer-impl
bajtos Sep 16, 2024
778f3e9
cleanup
bajtos Sep 16, 2024
d14a7f0
feat: repository support for REST API
bajtos Sep 16, 2024
89ec9b2
Merge branch 'main' into indexer-impl
bajtos Sep 16, 2024
a0a153a
ci: setup Redis for tests
bajtos Sep 16, 2024
1e31450
test: remove race condition in the timeout test
bajtos Sep 16, 2024
cafe142
feat: deploy to Fly.io
bajtos Sep 16, 2024
455b1ae
perf: remember walker state between steps
bajtos Sep 18, 2024
d89d33a
Update indexer/bin/piece-indexer.js
bajtos Sep 18, 2024
0755f2a
Update indexer/bin/piece-indexer.js
bajtos Sep 18, 2024
fa18f10
fixup! providerIdsBeingWalked
bajtos Sep 18, 2024
052ed7a
fixup! last_head -> lastHead
bajtos Sep 18, 2024
cc70be0
fixup! fix walker state reuse
bajtos Sep 18, 2024
faaa285
Update indexer/package.json
bajtos Sep 19, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
fly.toml
Dockerfile
.dockerignore
node_modules
*/node_modules
.git
.env
54 changes: 54 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,63 @@ jobs:
- uses: actions/setup-node@v4
with:
node-version: 20
- uses: supercharge/redis-github-action@1.8.0
- run: npm ci
- run: npm test

docker-build:
runs-on: ubuntu-latest
permissions:
contents: read
packages: read
env:
REGISTRY: ghcr.io
steps:
- uses: actions/checkout@v4

- uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Build monorepo Docker image
uses: docker/build-push-action@v6
with:
context: .
cache-from: type=registry,ref=ghcr.io/filecoin-station/piece-indexer
cache-to: type=inline

deploy-indexer:
if: github.ref == 'refs/heads/main'
needs: [test-indexer, docker-build]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: superfly/flyctl-actions/setup-flyctl@master
- run: flyctl deploy --remote-only -c api/fly.toml
env:
FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN_INDEXER }}
- if: failure()
uses: slackapi/slack-github-action@v1.27.0
with:
channel-id: alerts
payload: |
{
"text": "Deployment of `${{ github.event.repository.name }}` failed",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": ":warning: *<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|Deployment of `${{ github.event.repository.name }}` failed>*"
}
}
]
}
env:
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}

lint-all:
runs-on: ubuntu-latest
steps:
Expand Down
23 changes: 23 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# syntax = docker/dockerfile:1

FROM node:20.13.1-slim AS base
LABEL fly_launch_runtime="nodejs"
WORKDIR /app
ENV NODE_ENV=production

# Throw-away build stage to reduce size of final image
FROM base AS build

COPY --link package-lock.json package.json ./
# We cannot use a wildcard until `COPY --parents` is stabilised
# See https://docs.docker.com/reference/dockerfile/#copy---parents
COPY --link indexer/package.json ./indexer/
RUN npm ci --workspaces
COPY --link . .

# Final stage for app image
FROM base
COPY --from=build /app /app
# This argument controls the value used by npm to choose which workspace (subdir) to start
ENV NPM_CONFIG_WORKSPACE=""
CMD [ "npm", "start" ]
19 changes: 19 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,22 @@
A lightweight IPNI node mapping Filecoin PieceCID → payload block CID.

- [Design doc](./docs/design.md)

## Development

```bash
docker run --name redis -p 6379:6379 -d redis
npm start -w indexer
```

## Deployment

Pushes to `main` will be deployed automatically.

Perform manual devops using [Fly.io](https://fly.io):

Indexer:

```bash
$ fly deploy --remote-only -c indexer/fly.toml
```
Loading