Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
418.406748ms trace:
https://jaeger.dev.storj.io/trace/56fab37446cd7b15
FAIL (2025-12-03 08:49:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 418.431591ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
445.308569ms trace:
https://jaeger.dev.storj.io/trace/56fab37446cd7b15
FAIL (2025-12-03 08:48:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 445.331873ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
483.407141ms trace:
https://jaeger.dev.storj.io/trace/56fab37446cd7b15
FAIL (2025-12-03 08:47:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 483.436752ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
434.178765ms trace:
https://jaeger.dev.storj.io/trace/56fab37446cd7b15
FAIL (2025-12-03 08:46:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 434.19924ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
452.65351ms trace:
https://jaeger.dev.storj.io/trace/56fab37446cd7b15
FAIL (2025-12-03 08:45:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 452.676938ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
416.699383ms trace:
https://jaeger.dev.storj.io/trace/56fab37446cd7b15
FAIL (2025-12-03 08:45:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 416.721857ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
441.646939ms trace:
https://jaeger.dev.storj.io/trace/56fab37446cd7b15
FAIL (2025-12-03 08:44:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 441.677903ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
417.272118ms trace:
https://jaeger.dev.storj.io/trace/56fab37446cd7b15
FAIL (2025-12-03 08:43:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 417.350813ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
444.143973ms trace:
https://jaeger.dev.storj.io/trace/56fab37446cd7b15
FAIL (2025-12-03 08:42:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 444.164574ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
447.983823ms trace:
https://jaeger.dev.storj.io/trace/56fab37446cd7b15
FAIL (2025-12-03 08:41:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 448.00784ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
443.399166ms trace:
https://jaeger.dev.storj.io/trace/56fab37446cd7b15
FAIL (2025-12-03 08:40:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 443.468107ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.001702211s trace:
https://jaeger.dev.storj.io/trace/56fab37446cd7b15
FAIL (2025-12-03 08:40:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001742611s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
385.308162ms trace:
https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7
FAIL (2025-12-03 03:49:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload
Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload
Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments...
Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false
Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment...
Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment.
Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment
Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments...
Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded
Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op
Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments...
Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 385.330699ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
418.337504ms trace:
https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7
FAIL (2025-12-03 03:48:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload
Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload
Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments...
Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false
Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment...
Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment.
Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment
Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments...
Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded
Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op
Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments...
Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 418.358399ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [X...........................................................] duration:
421.911011ms trace:
https://jaeger.dev.storj.io/trace/2a81d3faba8ce874
FAIL (2025-12-03 03:47:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 422.612089ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
402.094117ms trace:
https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7
FAIL (2025-12-03 03:47:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload
Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload
Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments...
Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false
Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment...
Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment.
Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment
Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments...
Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded
Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op
Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments...
Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 402.122062ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
508.241464ms trace:
https://jaeger.dev.storj.io/trace/6d52f5e384db403b
FAIL (2025-12-03 03:47:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
t next segment. Inline: false
Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment...
Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment.
Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment
Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments...
Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments...
Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 508.268095ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.X..........................................................] duration:
443.656556ms trace:
https://jaeger.dev.storj.io/trace/2a81d3faba8ce874
FAIL (2025-12-03 03:47:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 443.678505ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
372.88848ms trace:
https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7
FAIL (2025-12-03 03:46:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload
Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload
Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments...
Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false
Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment...
Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment.
Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment
Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments...
Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded
Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op
Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments...
Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 372.91376ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
521.861124ms trace:
https://jaeger.dev.storj.io/trace/6d52f5e384db403b
FAIL (2025-12-03 03:46:28 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
t next segment. Inline: false
Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment...
Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment.
Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment
Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments...
Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments...
Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 521.959452ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
465.331445ms trace:
https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7
FAIL (2025-12-03 03:46:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload
Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload
Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments...
Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false
Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment...
Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment.
Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment
Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments...
Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded
Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op
Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments...
Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 465.369871ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..X.........................................................] duration:
446.731513ms trace:
https://jaeger.dev.storj.io/trace/2a81d3faba8ce874
FAIL (2025-12-03 03:45:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 446.756873ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
464.136755ms trace:
https://jaeger.dev.storj.io/trace/6d52f5e384db403b
FAIL (2025-12-03 03:45:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
t next segment. Inline: false
Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment...
Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment.
Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment
Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments...
Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments...
Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 464.164067ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [X..X........................................................] duration:
445.268091ms trace:
https://jaeger.dev.storj.io/trace/2a81d3faba8ce874
FAIL (2025-12-03 03:44:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 445.33094ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
372.763129ms trace:
https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7
FAIL (2025-12-03 03:44:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload
Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload
Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments...
Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false
Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment...
Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment.
Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment
Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments...
Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded
Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op
Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments...
Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 372.790775ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
513.611978ms trace:
https://jaeger.dev.storj.io/trace/6d52f5e384db403b
FAIL (2025-12-03 03:44:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
t next segment. Inline: false
Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment...
Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment.
Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment
Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments...
Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments...
Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 513.642247ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
464.953024ms trace:
https://jaeger.dev.storj.io/trace/6d52f5e384db403b
FAIL (2025-12-03 03:44:16 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
t next segment. Inline: false
Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment...
Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment.
Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment
Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments...
Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments...
Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 464.979302ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [XX..X.......................................................] duration:
679.242365ms trace:
https://jaeger.dev.storj.io/trace/2a81d3faba8ce874
FAIL (2025-12-03 03:44:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 679.268637ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
404.867255ms trace:
https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7
FAIL (2025-12-03 03:43:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload
Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload
Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments...
Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false
Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment...
Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment.
Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment
Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments...
Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded
Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op
Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments...
Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 404.926846ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
955.103044ms trace:
https://jaeger.dev.storj.io/trace/6d52f5e384db403b
FAIL (2025-12-03 03:43:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
t next segment. Inline: false
Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment...
Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment.
Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment
Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments...
Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments...
Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 955.127125ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [.XX..X......................................................] duration:
448.485438ms trace:
https://jaeger.dev.storj.io/trace/2a81d3faba8ce874
FAIL (2025-12-03 03:43:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 448.507674ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [..XX..X.....................................................] duration:
425.065633ms trace:
https://jaeger.dev.storj.io/trace/2a81d3faba8ce874
FAIL (2025-12-03 03:43:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 425.152754ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
405.891392ms trace:
https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7
FAIL (2025-12-03 03:42:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload
Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload
Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments...
Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false
Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment...
Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment.
Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment
Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments...
Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded
Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op
Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments...
Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 405.907566ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
522.550189ms trace:
https://jaeger.dev.storj.io/trace/6d52f5e384db403b
FAIL (2025-12-03 03:42:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
t next segment. Inline: false
Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment...
Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment.
Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment
Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments...
Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments...
Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 522.574058ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [...XX..X....................................................] duration:
591.865663ms trace:
https://jaeger.dev.storj.io/trace/2a81d3faba8ce874
FAIL (2025-12-03 03:42:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 591.952359ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [....XX..X...................................................] duration:
446.846225ms trace:
https://jaeger.dev.storj.io/trace/2a81d3faba8ce874
FAIL (2025-12-03 03:42:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 446.870601ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
421.335813ms trace:
https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7
FAIL (2025-12-03 03:42:16 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload
Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload
Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments...
Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false
Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment...
Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment.
Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment
Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments...
Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded
Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op
Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments...
Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 421.37613ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [.....XX..X..................................................] duration:
432.716703ms trace:
https://jaeger.dev.storj.io/trace/2a81d3faba8ce874
FAIL (2025-12-03 03:42:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 432.74348ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
484.404022ms trace:
https://jaeger.dev.storj.io/trace/6d52f5e384db403b
FAIL (2025-12-03 03:41:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
t next segment. Inline: false
Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment...
Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment.
Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment
Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments...
Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments...
Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 484.43233ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [......XX..X.................................................] duration:
437.964598ms trace:
https://jaeger.dev.storj.io/trace/2a81d3faba8ce874
FAIL (2025-12-03 03:41:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 437.99598ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
382.272476ms trace:
https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7
FAIL (2025-12-03 03:41:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload
Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload
Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments...
Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false
Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment...
Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment.
Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment
Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments...
Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded
Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op
Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments...
Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 382.297501ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
444.589461ms trace:
https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7
FAIL (2025-12-03 03:40:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload
Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload
Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments...
Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false
Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment...
Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment.
Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment
Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments...
Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded
Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op
Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments...
Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 444.620144ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [.......XX..X................................................] duration:
445.598436ms trace:
https://jaeger.dev.storj.io/trace/2a81d3faba8ce874
FAIL (2025-12-03 03:40:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 445.627028ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
456.005074ms trace:
https://jaeger.dev.storj.io/trace/6d52f5e384db403b
FAIL (2025-12-03 03:40:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
t next segment. Inline: false
Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment...
Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment.
Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment
Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments...
Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments...
Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 456.028468ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [........XX..X...............................................] duration:
451.164614ms trace:
https://jaeger.dev.storj.io/trace/2a81d3faba8ce874
FAIL (2025-12-03 03:40:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 451.195768ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
425.894368ms trace:
https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7
FAIL (2025-12-03 03:40:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload
Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload
Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments...
Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false
Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment...
Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment.
Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment
Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments...
Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded
Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op
Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments...
Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 425.920937ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
524.959544ms trace:
https://jaeger.dev.storj.io/trace/6d52f5e384db403b
FAIL (2025-12-03 03:39:46 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
t next segment. Inline: false
Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment...
Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment.
Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment
Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments...
Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments...
Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 524.987586ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [.........XX..X..............................................] duration:
471.524663ms trace:
https://jaeger.dev.storj.io/trace/2a81d3faba8ce874
FAIL (2025-12-03 03:39:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 471.586982ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
520.972882ms trace:
https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7
FAIL (2025-12-03 03:39:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload
Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload
Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments...
Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false
Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment...
Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment.
Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece...
Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment
Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments...
Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil>
Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded
Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op
Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments...
Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 521.032494ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
419.984096ms trace:
https://jaeger.dev.storj.io/trace/6d52f5e384db403b
FAIL (2025-12-03 03:39:16 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
t next segment. Inline: false
Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment...
Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment.
Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece...
Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment
Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments...
Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil>
Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments...
Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 420.008695ms
core
uplink
us-select-1
us1-placement-20
|