Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 148722 reports (Page 70 of 2975)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....X.....................................X.................] duration: 460.188639ms trace: https://jaeger.dev.storj.io/trace/49e5f4c0e8587b3d FAIL (2025-10-31 22:07:00 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 460.4204ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 549.621412ms trace: https://jaeger.dev.storj.io/trace/25b2c21daa3d085 FAIL (2025-10-31 22:06:43 UTC) orbiter warning: unexpected EOF Duration: 549.790261ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.....X..................................................] duration: 387.961386ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 22:06:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 387.985937ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....X.....................................X................] duration: 735.814358ms trace: https://jaeger.dev.storj.io/trace/49e5f4c0e8587b3d FAIL (2025-10-31 22:06:40 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 735.83742ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X......................................X..................] duration: 504.53313ms trace: https://jaeger.dev.storj.io/trace/41c2b066eb9ea5e8 FAIL (2025-10-31 22:06:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.243590254 (34h14m25.907539233s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244078634 (34h14m25.908027607s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244974253 (34h14m25.908923236s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.245239075 (34h14m25.909188054s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246191747 (34h14m25.910140732s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246721069 (34h14m25.910670051s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247101943 (34h14m25.911050925s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247367511 (34h14m25.911316501s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247624227 (34h14m25.911573229s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247887850 (34h14m25.911836838s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248392598 (34h14m25.912341583s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248921469 (34h14m25.912870457s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249532021 (34h14m25.913481008s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249926757 (34h14m25.913875738s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250408522 (34h14m25.91435751s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250881171 (34h14m25.914830153s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.251325874 (34h14m25.915274858s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.319899795 (34h14m25.983848763s) [upload=35594]: Next returned nil segment Oct 31 21:53:55.319912385 (34h14m25.983861365s) [upload=35594]: Waiting for error group managing segments... Oct 31 21:53:55.332186509 (34h14m25.996135498s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.337196032 (34h14m26.001145012s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.345822268 (34h14m26.009771248s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.356624949 (34h14m26.020573918s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.357992611 (34h14m26.021941606s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.358284965 (34h14m26.022233957s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.363582356 (34h14m26.027531346s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.376206640 (34h14m26.040155641s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.386414599 (34h14m26.050363579s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.030428990 (34h15m20.694377969s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.506293399 (34h15m21.170242379s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:55.184164260 (34h15m25.848113246s) [noise=true , address=45.91.116.14:4203 , piece_id=DXLZT7TNAP7BZXQU535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection Oct 31 21:54:55.184248337 (34h15m25.848197346s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:187b9r6f2MK5c5mJEVasrhfpRHzhC4JbkWTbVvHuqB62XYs8xL, address:77.90.31.13:4641): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184319241 (34h15m25.84826822s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection Oct 31 21:54:55.184492039 (34h15m25.848441022s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184662449 (34h15m25.848611434s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: read tcp 159.223.143.39:38358->45.91.116.4:4964: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184739659 (34h15m25.848688638s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: write tcp 159.223.143.39:41986->77.90.31.14:4686: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184756157 (34h15m25.848705142s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1xwaFojWAo54xD34ZA7HACYmJf811KFqN9biJJoag5oxE9fJqx, address:45.91.116.13:4177): protocol: read tcp 159.223.143.39:41676->45.91.116.13:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184786251 (34h15m25.848735222s) [seg_pos={0 0}, upload=35594]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:54:55.206244634 (34h15m25.870193618s) [upload=35594]: Done uploading segments... Oct 31 21:54:55.206282843 (34h15m25.870231833s) [upload=35594]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 504.586294ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......X.....................................X...............] duration: 468.690798ms trace: https://jaeger.dev.storj.io/trace/49e5f4c0e8587b3d FAIL (2025-10-31 22:06:24 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 468.875281ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 551.460643ms trace: https://jaeger.dev.storj.io/trace/25b2c21daa3d085 FAIL (2025-10-31 22:06:22 UTC) orbiter warning: unexpected EOF Duration: 551.486283ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X......................................X..............] duration: 453.109336ms trace: https://jaeger.dev.storj.io/trace/3adce5f38013470e FAIL (2025-10-31 22:06:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 453.135522ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.................................X................] duration: 395.345314ms trace: https://jaeger.dev.storj.io/trace/236699c64467bb84 FAIL (2025-10-31 22:06:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:54.316715391 (34h14m24.980664371s) [upload=35593]: Starting upload Oct 31 21:53:54.316916021 (34h14m24.980865001s) [upload=35593]: Done starting upload Oct 31 21:53:54.316999602 (34h14m24.980948574s) [upload=35593]: Uploading segments... Oct 31 21:53:54.317521796 (34h14m24.981470776s) [upload=35593]: Got next segment. Inline: false Oct 31 21:53:54.317593157 (34h14m24.98154214s) [upload=35593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:54.317600463 (34h14m24.981549444s) [upload=35593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:54.344972306 (34h14m25.008921293s) [seg_pos={0 0}, upload=35593]: Begin upload segment... Oct 31 21:53:54.345030267 (34h14m25.008979247s) [seg_pos={0 0}, upload=35593]: Done begin upload segment. Oct 31 21:53:54.345213856 (34h14m25.009162844s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345225337 (34h14m25.009174327s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345830005 (34h14m25.009778994s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346300953 (34h14m25.010249931s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346533214 (34h14m25.010482193s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347118612 (34h14m25.011067594s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347365203 (34h14m25.011314187s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347568970 (34h14m25.01151796s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347824532 (34h14m25.01177352s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347959678 (34h14m25.011908664s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348214346 (34h14m25.012163334s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348367787 (34h14m25.012316775s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348778049 (34h14m25.012727028s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349186865 (34h14m25.013135844s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349752289 (34h14m25.013701272s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.352239055 (34h14m25.016188036s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.419826216 (34h14m25.083775196s) [upload=35593]: Next returned nil segment Oct 31 21:53:54.419839894 (34h14m25.083788873s) [upload=35593]: Waiting for error group managing segments... Oct 31 21:53:54.425324284 (34h14m25.089273265s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426679856 (34h14m25.090628835s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426719775 (34h14m25.090668754s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427269718 (34h14m25.091218697s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427363119 (34h14m25.0913121s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427761264 (34h14m25.091710245s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428214622 (34h14m25.092163602s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428234634 (34h14m25.092183614s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428729183 (34h14m25.092678166s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428832310 (34h14m25.092781292s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.429592128 (34h14m25.093541127s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.447797690 (34h14m25.111746672s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:54:24.282494056 (34h14m54.946443036s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: read tcp 159.223.143.39:35840->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282509279 (34h14m54.946458269s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 159.223.143.39:36998->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282717367 (34h14m54.946666355s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 159.223.143.39:55516->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282833181 (34h14m54.946782153s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 159.223.143.39:35952->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282879533 (34h14m54.946828499s) [seg_pos={0 0}, upload=35593]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:24.306226453 (34h14m54.970175433s) [upload=35593]: Done uploading segments... Oct 31 21:54:24.306271889 (34h14m54.970220869s) [upload=35593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 395.371503ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.....X.................................................] duration: 441.698424ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 22:06:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 441.728455ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X......................................X.................] duration: 567.910827ms trace: https://jaeger.dev.storj.io/trace/41c2b066eb9ea5e8 FAIL (2025-10-31 22:05:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.243590254 (34h14m25.907539233s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244078634 (34h14m25.908027607s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244974253 (34h14m25.908923236s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.245239075 (34h14m25.909188054s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246191747 (34h14m25.910140732s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246721069 (34h14m25.910670051s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247101943 (34h14m25.911050925s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247367511 (34h14m25.911316501s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247624227 (34h14m25.911573229s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247887850 (34h14m25.911836838s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248392598 (34h14m25.912341583s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248921469 (34h14m25.912870457s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249532021 (34h14m25.913481008s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249926757 (34h14m25.913875738s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250408522 (34h14m25.91435751s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250881171 (34h14m25.914830153s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.251325874 (34h14m25.915274858s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.319899795 (34h14m25.983848763s) [upload=35594]: Next returned nil segment Oct 31 21:53:55.319912385 (34h14m25.983861365s) [upload=35594]: Waiting for error group managing segments... Oct 31 21:53:55.332186509 (34h14m25.996135498s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.337196032 (34h14m26.001145012s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.345822268 (34h14m26.009771248s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.356624949 (34h14m26.020573918s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.357992611 (34h14m26.021941606s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.358284965 (34h14m26.022233957s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.363582356 (34h14m26.027531346s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.376206640 (34h14m26.040155641s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.386414599 (34h14m26.050363579s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.030428990 (34h15m20.694377969s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.506293399 (34h15m21.170242379s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:55.184164260 (34h15m25.848113246s) [noise=true , address=45.91.116.14:4203 , piece_id=DXLZT7TNAP7BZXQU535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection Oct 31 21:54:55.184248337 (34h15m25.848197346s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:187b9r6f2MK5c5mJEVasrhfpRHzhC4JbkWTbVvHuqB62XYs8xL, address:77.90.31.13:4641): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184319241 (34h15m25.84826822s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection Oct 31 21:54:55.184492039 (34h15m25.848441022s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184662449 (34h15m25.848611434s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: read tcp 159.223.143.39:38358->45.91.116.4:4964: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184739659 (34h15m25.848688638s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: write tcp 159.223.143.39:41986->77.90.31.14:4686: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184756157 (34h15m25.848705142s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1xwaFojWAo54xD34ZA7HACYmJf811KFqN9biJJoag5oxE9fJqx, address:45.91.116.13:4177): protocol: read tcp 159.223.143.39:41676->45.91.116.13:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184786251 (34h15m25.848735222s) [seg_pos={0 0}, upload=35594]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:54:55.206244634 (34h15m25.870193618s) [upload=35594]: Done uploading segments... Oct 31 21:54:55.206282843 (34h15m25.870231833s) [upload=35594]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 567.949445ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......X.....................................X..............] duration: 572.492903ms trace: https://jaeger.dev.storj.io/trace/49e5f4c0e8587b3d FAIL (2025-10-31 22:05:28 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 572.52081ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 622.251364ms trace: https://jaeger.dev.storj.io/trace/25b2c21daa3d085 FAIL (2025-10-31 22:05:21 UTC) orbiter warning: unexpected EOF Duration: 622.276125ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.....X................................................] duration: 454.443936ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 22:05:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 454.470132ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.................................X...............] duration: 528.555427ms trace: https://jaeger.dev.storj.io/trace/236699c64467bb84 FAIL (2025-10-31 22:05:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:54.316715391 (34h14m24.980664371s) [upload=35593]: Starting upload Oct 31 21:53:54.316916021 (34h14m24.980865001s) [upload=35593]: Done starting upload Oct 31 21:53:54.316999602 (34h14m24.980948574s) [upload=35593]: Uploading segments... Oct 31 21:53:54.317521796 (34h14m24.981470776s) [upload=35593]: Got next segment. Inline: false Oct 31 21:53:54.317593157 (34h14m24.98154214s) [upload=35593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:54.317600463 (34h14m24.981549444s) [upload=35593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:54.344972306 (34h14m25.008921293s) [seg_pos={0 0}, upload=35593]: Begin upload segment... Oct 31 21:53:54.345030267 (34h14m25.008979247s) [seg_pos={0 0}, upload=35593]: Done begin upload segment. Oct 31 21:53:54.345213856 (34h14m25.009162844s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345225337 (34h14m25.009174327s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345830005 (34h14m25.009778994s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346300953 (34h14m25.010249931s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346533214 (34h14m25.010482193s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347118612 (34h14m25.011067594s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347365203 (34h14m25.011314187s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347568970 (34h14m25.01151796s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347824532 (34h14m25.01177352s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347959678 (34h14m25.011908664s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348214346 (34h14m25.012163334s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348367787 (34h14m25.012316775s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348778049 (34h14m25.012727028s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349186865 (34h14m25.013135844s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349752289 (34h14m25.013701272s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.352239055 (34h14m25.016188036s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.419826216 (34h14m25.083775196s) [upload=35593]: Next returned nil segment Oct 31 21:53:54.419839894 (34h14m25.083788873s) [upload=35593]: Waiting for error group managing segments... Oct 31 21:53:54.425324284 (34h14m25.089273265s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426679856 (34h14m25.090628835s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426719775 (34h14m25.090668754s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427269718 (34h14m25.091218697s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427363119 (34h14m25.0913121s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427761264 (34h14m25.091710245s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428214622 (34h14m25.092163602s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428234634 (34h14m25.092183614s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428729183 (34h14m25.092678166s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428832310 (34h14m25.092781292s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.429592128 (34h14m25.093541127s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.447797690 (34h14m25.111746672s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:54:24.282494056 (34h14m54.946443036s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: read tcp 159.223.143.39:35840->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282509279 (34h14m54.946458269s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 159.223.143.39:36998->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282717367 (34h14m54.946666355s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 159.223.143.39:55516->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282833181 (34h14m54.946782153s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 159.223.143.39:35952->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282879533 (34h14m54.946828499s) [seg_pos={0 0}, upload=35593]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:24.306226453 (34h14m54.970175433s) [upload=35593]: Done uploading segments... Oct 31 21:54:24.306271889 (34h14m54.970220869s) [upload=35593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 528.577136ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X......................................X.............] duration: 662.565227ms trace: https://jaeger.dev.storj.io/trace/3adce5f38013470e FAIL (2025-10-31 22:05:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 662.58912ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.................................X..............] duration: 455.940118ms trace: https://jaeger.dev.storj.io/trace/236699c64467bb84 FAIL (2025-10-31 22:04:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:54.316715391 (34h14m24.980664371s) [upload=35593]: Starting upload Oct 31 21:53:54.316916021 (34h14m24.980865001s) [upload=35593]: Done starting upload Oct 31 21:53:54.316999602 (34h14m24.980948574s) [upload=35593]: Uploading segments... Oct 31 21:53:54.317521796 (34h14m24.981470776s) [upload=35593]: Got next segment. Inline: false Oct 31 21:53:54.317593157 (34h14m24.98154214s) [upload=35593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:54.317600463 (34h14m24.981549444s) [upload=35593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:54.344972306 (34h14m25.008921293s) [seg_pos={0 0}, upload=35593]: Begin upload segment... Oct 31 21:53:54.345030267 (34h14m25.008979247s) [seg_pos={0 0}, upload=35593]: Done begin upload segment. Oct 31 21:53:54.345213856 (34h14m25.009162844s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345225337 (34h14m25.009174327s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345830005 (34h14m25.009778994s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346300953 (34h14m25.010249931s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346533214 (34h14m25.010482193s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347118612 (34h14m25.011067594s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347365203 (34h14m25.011314187s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347568970 (34h14m25.01151796s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347824532 (34h14m25.01177352s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347959678 (34h14m25.011908664s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348214346 (34h14m25.012163334s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348367787 (34h14m25.012316775s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348778049 (34h14m25.012727028s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349186865 (34h14m25.013135844s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349752289 (34h14m25.013701272s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.352239055 (34h14m25.016188036s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.419826216 (34h14m25.083775196s) [upload=35593]: Next returned nil segment Oct 31 21:53:54.419839894 (34h14m25.083788873s) [upload=35593]: Waiting for error group managing segments... Oct 31 21:53:54.425324284 (34h14m25.089273265s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426679856 (34h14m25.090628835s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426719775 (34h14m25.090668754s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427269718 (34h14m25.091218697s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427363119 (34h14m25.0913121s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427761264 (34h14m25.091710245s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428214622 (34h14m25.092163602s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428234634 (34h14m25.092183614s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428729183 (34h14m25.092678166s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428832310 (34h14m25.092781292s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.429592128 (34h14m25.093541127s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.447797690 (34h14m25.111746672s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:54:24.282494056 (34h14m54.946443036s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: read tcp 159.223.143.39:35840->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282509279 (34h14m54.946458269s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 159.223.143.39:36998->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282717367 (34h14m54.946666355s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 159.223.143.39:55516->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282833181 (34h14m54.946782153s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 159.223.143.39:35952->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282879533 (34h14m54.946828499s) [seg_pos={0 0}, upload=35593]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:24.306226453 (34h14m54.970175433s) [upload=35593]: Done uploading segments... Oct 31 21:54:24.306271889 (34h14m54.970220869s) [upload=35593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 455.973812ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.....X...............................................] duration: 382.878688ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 22:04:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 382.912183ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [X.......X.....................................X.............] duration: 588.016429ms trace: https://jaeger.dev.storj.io/trace/49e5f4c0e8587b3d FAIL (2025-10-31 22:04:28 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 588.031845ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.......X......................................X............] duration: 466.08573ms trace: https://jaeger.dev.storj.io/trace/3adce5f38013470e FAIL (2025-10-31 22:04:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 466.110716ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X......................................X................] duration: 546.746004ms trace: https://jaeger.dev.storj.io/trace/41c2b066eb9ea5e8 FAIL (2025-10-31 22:04:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.243590254 (34h14m25.907539233s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244078634 (34h14m25.908027607s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244974253 (34h14m25.908923236s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.245239075 (34h14m25.909188054s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246191747 (34h14m25.910140732s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246721069 (34h14m25.910670051s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247101943 (34h14m25.911050925s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247367511 (34h14m25.911316501s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247624227 (34h14m25.911573229s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247887850 (34h14m25.911836838s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248392598 (34h14m25.912341583s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248921469 (34h14m25.912870457s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249532021 (34h14m25.913481008s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249926757 (34h14m25.913875738s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250408522 (34h14m25.91435751s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250881171 (34h14m25.914830153s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.251325874 (34h14m25.915274858s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.319899795 (34h14m25.983848763s) [upload=35594]: Next returned nil segment Oct 31 21:53:55.319912385 (34h14m25.983861365s) [upload=35594]: Waiting for error group managing segments... Oct 31 21:53:55.332186509 (34h14m25.996135498s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.337196032 (34h14m26.001145012s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.345822268 (34h14m26.009771248s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.356624949 (34h14m26.020573918s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.357992611 (34h14m26.021941606s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.358284965 (34h14m26.022233957s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.363582356 (34h14m26.027531346s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.376206640 (34h14m26.040155641s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.386414599 (34h14m26.050363579s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.030428990 (34h15m20.694377969s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.506293399 (34h15m21.170242379s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:55.184164260 (34h15m25.848113246s) [noise=true , address=45.91.116.14:4203 , piece_id=DXLZT7TNAP7BZXQU535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection Oct 31 21:54:55.184248337 (34h15m25.848197346s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:187b9r6f2MK5c5mJEVasrhfpRHzhC4JbkWTbVvHuqB62XYs8xL, address:77.90.31.13:4641): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184319241 (34h15m25.84826822s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection Oct 31 21:54:55.184492039 (34h15m25.848441022s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184662449 (34h15m25.848611434s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: read tcp 159.223.143.39:38358->45.91.116.4:4964: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184739659 (34h15m25.848688638s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: write tcp 159.223.143.39:41986->77.90.31.14:4686: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184756157 (34h15m25.848705142s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1xwaFojWAo54xD34ZA7HACYmJf811KFqN9biJJoag5oxE9fJqx, address:45.91.116.13:4177): protocol: read tcp 159.223.143.39:41676->45.91.116.13:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184786251 (34h15m25.848735222s) [seg_pos={0 0}, upload=35594]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:54:55.206244634 (34h15m25.870193618s) [upload=35594]: Done uploading segments... Oct 31 21:54:55.206282843 (34h15m25.870231833s) [upload=35594]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 546.772788ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [X.............................................X.............] duration: 637.561497ms trace: https://jaeger.dev.storj.io/trace/25b2c21daa3d085 FAIL (2025-10-31 22:04:06 UTC) orbiter warning: unexpected EOF Duration: 637.593567ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X......................................X...............] duration: 565.675833ms trace: https://jaeger.dev.storj.io/trace/41c2b066eb9ea5e8 FAIL (2025-10-31 22:03:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.243590254 (34h14m25.907539233s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244078634 (34h14m25.908027607s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244974253 (34h14m25.908923236s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.245239075 (34h14m25.909188054s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246191747 (34h14m25.910140732s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246721069 (34h14m25.910670051s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247101943 (34h14m25.911050925s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247367511 (34h14m25.911316501s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247624227 (34h14m25.911573229s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247887850 (34h14m25.911836838s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248392598 (34h14m25.912341583s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248921469 (34h14m25.912870457s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249532021 (34h14m25.913481008s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249926757 (34h14m25.913875738s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250408522 (34h14m25.91435751s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250881171 (34h14m25.914830153s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.251325874 (34h14m25.915274858s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.319899795 (34h14m25.983848763s) [upload=35594]: Next returned nil segment Oct 31 21:53:55.319912385 (34h14m25.983861365s) [upload=35594]: Waiting for error group managing segments... Oct 31 21:53:55.332186509 (34h14m25.996135498s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.337196032 (34h14m26.001145012s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.345822268 (34h14m26.009771248s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.356624949 (34h14m26.020573918s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.357992611 (34h14m26.021941606s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.358284965 (34h14m26.022233957s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.363582356 (34h14m26.027531346s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.376206640 (34h14m26.040155641s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.386414599 (34h14m26.050363579s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.030428990 (34h15m20.694377969s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.506293399 (34h15m21.170242379s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:55.184164260 (34h15m25.848113246s) [noise=true , address=45.91.116.14:4203 , piece_id=DXLZT7TNAP7BZXQU535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection Oct 31 21:54:55.184248337 (34h15m25.848197346s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:187b9r6f2MK5c5mJEVasrhfpRHzhC4JbkWTbVvHuqB62XYs8xL, address:77.90.31.13:4641): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184319241 (34h15m25.84826822s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection Oct 31 21:54:55.184492039 (34h15m25.848441022s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184662449 (34h15m25.848611434s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: read tcp 159.223.143.39:38358->45.91.116.4:4964: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184739659 (34h15m25.848688638s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: write tcp 159.223.143.39:41986->77.90.31.14:4686: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184756157 (34h15m25.848705142s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1xwaFojWAo54xD34ZA7HACYmJf811KFqN9biJJoag5oxE9fJqx, address:45.91.116.13:4177): protocol: read tcp 159.223.143.39:41676->45.91.116.13:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184786251 (34h15m25.848735222s) [seg_pos={0 0}, upload=35594]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:54:55.206244634 (34h15m25.870193618s) [upload=35594]: Done uploading segments... Oct 31 21:54:55.206282843 (34h15m25.870231833s) [upload=35594]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 565.705314ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.................................X.............] duration: 578.58086ms trace: https://jaeger.dev.storj.io/trace/236699c64467bb84 FAIL (2025-10-31 22:03:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:54.316715391 (34h14m24.980664371s) [upload=35593]: Starting upload Oct 31 21:53:54.316916021 (34h14m24.980865001s) [upload=35593]: Done starting upload Oct 31 21:53:54.316999602 (34h14m24.980948574s) [upload=35593]: Uploading segments... Oct 31 21:53:54.317521796 (34h14m24.981470776s) [upload=35593]: Got next segment. Inline: false Oct 31 21:53:54.317593157 (34h14m24.98154214s) [upload=35593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:54.317600463 (34h14m24.981549444s) [upload=35593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:54.344972306 (34h14m25.008921293s) [seg_pos={0 0}, upload=35593]: Begin upload segment... Oct 31 21:53:54.345030267 (34h14m25.008979247s) [seg_pos={0 0}, upload=35593]: Done begin upload segment. Oct 31 21:53:54.345213856 (34h14m25.009162844s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345225337 (34h14m25.009174327s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345830005 (34h14m25.009778994s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346300953 (34h14m25.010249931s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346533214 (34h14m25.010482193s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347118612 (34h14m25.011067594s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347365203 (34h14m25.011314187s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347568970 (34h14m25.01151796s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347824532 (34h14m25.01177352s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347959678 (34h14m25.011908664s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348214346 (34h14m25.012163334s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348367787 (34h14m25.012316775s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348778049 (34h14m25.012727028s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349186865 (34h14m25.013135844s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349752289 (34h14m25.013701272s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.352239055 (34h14m25.016188036s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.419826216 (34h14m25.083775196s) [upload=35593]: Next returned nil segment Oct 31 21:53:54.419839894 (34h14m25.083788873s) [upload=35593]: Waiting for error group managing segments... Oct 31 21:53:54.425324284 (34h14m25.089273265s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426679856 (34h14m25.090628835s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426719775 (34h14m25.090668754s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427269718 (34h14m25.091218697s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427363119 (34h14m25.0913121s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427761264 (34h14m25.091710245s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428214622 (34h14m25.092163602s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428234634 (34h14m25.092183614s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428729183 (34h14m25.092678166s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428832310 (34h14m25.092781292s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.429592128 (34h14m25.093541127s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.447797690 (34h14m25.111746672s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:54:24.282494056 (34h14m54.946443036s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: read tcp 159.223.143.39:35840->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282509279 (34h14m54.946458269s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 159.223.143.39:36998->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282717367 (34h14m54.946666355s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 159.223.143.39:55516->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282833181 (34h14m54.946782153s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 159.223.143.39:35952->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282879533 (34h14m54.946828499s) [seg_pos={0 0}, upload=35593]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:24.306226453 (34h14m54.970175433s) [upload=35593]: Done uploading segments... Oct 31 21:54:24.306271889 (34h14m54.970220869s) [upload=35593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 578.606253ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.......X......................................X...........] duration: 525.813136ms trace: https://jaeger.dev.storj.io/trace/3adce5f38013470e FAIL (2025-10-31 22:03:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 525.837739ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.....X..............................................] duration: 388.229246ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 22:03:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 388.256907ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.X.......X.....................................X............] duration: 522.384293ms trace: https://jaeger.dev.storj.io/trace/49e5f4c0e8587b3d FAIL (2025-10-31 22:03:28 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 522.408885ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X......................................X..............] duration: 529.677874ms trace: https://jaeger.dev.storj.io/trace/41c2b066eb9ea5e8 FAIL (2025-10-31 22:03:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.243590254 (34h14m25.907539233s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244078634 (34h14m25.908027607s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244974253 (34h14m25.908923236s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.245239075 (34h14m25.909188054s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246191747 (34h14m25.910140732s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246721069 (34h14m25.910670051s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247101943 (34h14m25.911050925s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247367511 (34h14m25.911316501s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247624227 (34h14m25.911573229s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247887850 (34h14m25.911836838s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248392598 (34h14m25.912341583s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248921469 (34h14m25.912870457s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249532021 (34h14m25.913481008s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249926757 (34h14m25.913875738s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250408522 (34h14m25.91435751s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250881171 (34h14m25.914830153s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.251325874 (34h14m25.915274858s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.319899795 (34h14m25.983848763s) [upload=35594]: Next returned nil segment Oct 31 21:53:55.319912385 (34h14m25.983861365s) [upload=35594]: Waiting for error group managing segments... Oct 31 21:53:55.332186509 (34h14m25.996135498s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.337196032 (34h14m26.001145012s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.345822268 (34h14m26.009771248s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.356624949 (34h14m26.020573918s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.357992611 (34h14m26.021941606s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.358284965 (34h14m26.022233957s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.363582356 (34h14m26.027531346s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.376206640 (34h14m26.040155641s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.386414599 (34h14m26.050363579s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.030428990 (34h15m20.694377969s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.506293399 (34h15m21.170242379s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:55.184164260 (34h15m25.848113246s) [noise=true , address=45.91.116.14:4203 , piece_id=DXLZT7TNAP7BZXQU535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection Oct 31 21:54:55.184248337 (34h15m25.848197346s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:187b9r6f2MK5c5mJEVasrhfpRHzhC4JbkWTbVvHuqB62XYs8xL, address:77.90.31.13:4641): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184319241 (34h15m25.84826822s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection Oct 31 21:54:55.184492039 (34h15m25.848441022s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184662449 (34h15m25.848611434s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: read tcp 159.223.143.39:38358->45.91.116.4:4964: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184739659 (34h15m25.848688638s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: write tcp 159.223.143.39:41986->77.90.31.14:4686: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184756157 (34h15m25.848705142s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1xwaFojWAo54xD34ZA7HACYmJf811KFqN9biJJoag5oxE9fJqx, address:45.91.116.13:4177): protocol: read tcp 159.223.143.39:41676->45.91.116.13:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184786251 (34h15m25.848735222s) [seg_pos={0 0}, upload=35594]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:54:55.206244634 (34h15m25.870193618s) [upload=35594]: Done uploading segments... Oct 31 21:54:55.206282843 (34h15m25.870231833s) [upload=35594]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 529.706288ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.X.............................................X............] duration: 581.783795ms trace: https://jaeger.dev.storj.io/trace/25b2c21daa3d085 FAIL (2025-10-31 22:03:11 UTC) orbiter warning: unexpected EOF Duration: 581.806607ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.................................X............] duration: 476.574828ms trace: https://jaeger.dev.storj.io/trace/236699c64467bb84 FAIL (2025-10-31 22:02:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:54.316715391 (34h14m24.980664371s) [upload=35593]: Starting upload Oct 31 21:53:54.316916021 (34h14m24.980865001s) [upload=35593]: Done starting upload Oct 31 21:53:54.316999602 (34h14m24.980948574s) [upload=35593]: Uploading segments... Oct 31 21:53:54.317521796 (34h14m24.981470776s) [upload=35593]: Got next segment. Inline: false Oct 31 21:53:54.317593157 (34h14m24.98154214s) [upload=35593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:54.317600463 (34h14m24.981549444s) [upload=35593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:54.344972306 (34h14m25.008921293s) [seg_pos={0 0}, upload=35593]: Begin upload segment... Oct 31 21:53:54.345030267 (34h14m25.008979247s) [seg_pos={0 0}, upload=35593]: Done begin upload segment. Oct 31 21:53:54.345213856 (34h14m25.009162844s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345225337 (34h14m25.009174327s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345830005 (34h14m25.009778994s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346300953 (34h14m25.010249931s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346533214 (34h14m25.010482193s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347118612 (34h14m25.011067594s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347365203 (34h14m25.011314187s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347568970 (34h14m25.01151796s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347824532 (34h14m25.01177352s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347959678 (34h14m25.011908664s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348214346 (34h14m25.012163334s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348367787 (34h14m25.012316775s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348778049 (34h14m25.012727028s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349186865 (34h14m25.013135844s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349752289 (34h14m25.013701272s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.352239055 (34h14m25.016188036s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.419826216 (34h14m25.083775196s) [upload=35593]: Next returned nil segment Oct 31 21:53:54.419839894 (34h14m25.083788873s) [upload=35593]: Waiting for error group managing segments... Oct 31 21:53:54.425324284 (34h14m25.089273265s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426679856 (34h14m25.090628835s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426719775 (34h14m25.090668754s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427269718 (34h14m25.091218697s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427363119 (34h14m25.0913121s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427761264 (34h14m25.091710245s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428214622 (34h14m25.092163602s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428234634 (34h14m25.092183614s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428729183 (34h14m25.092678166s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428832310 (34h14m25.092781292s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.429592128 (34h14m25.093541127s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.447797690 (34h14m25.111746672s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:54:24.282494056 (34h14m54.946443036s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: read tcp 159.223.143.39:35840->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282509279 (34h14m54.946458269s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 159.223.143.39:36998->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282717367 (34h14m54.946666355s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 159.223.143.39:55516->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282833181 (34h14m54.946782153s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 159.223.143.39:35952->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282879533 (34h14m54.946828499s) [seg_pos={0 0}, upload=35593]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:24.306226453 (34h14m54.970175433s) [upload=35593]: Done uploading segments... Oct 31 21:54:24.306271889 (34h14m54.970220869s) [upload=35593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 476.59671ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..X.............................................X...........] duration: 652.318691ms trace: https://jaeger.dev.storj.io/trace/25b2c21daa3d085 FAIL (2025-10-31 22:02:43 UTC) orbiter warning: unexpected EOF Duration: 652.344192ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X......................................X.............] duration: 564.484629ms trace: https://jaeger.dev.storj.io/trace/41c2b066eb9ea5e8 FAIL (2025-10-31 22:02:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.243590254 (34h14m25.907539233s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244078634 (34h14m25.908027607s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244974253 (34h14m25.908923236s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.245239075 (34h14m25.909188054s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246191747 (34h14m25.910140732s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246721069 (34h14m25.910670051s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247101943 (34h14m25.911050925s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247367511 (34h14m25.911316501s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247624227 (34h14m25.911573229s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247887850 (34h14m25.911836838s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248392598 (34h14m25.912341583s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248921469 (34h14m25.912870457s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249532021 (34h14m25.913481008s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249926757 (34h14m25.913875738s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250408522 (34h14m25.91435751s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250881171 (34h14m25.914830153s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.251325874 (34h14m25.915274858s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.319899795 (34h14m25.983848763s) [upload=35594]: Next returned nil segment Oct 31 21:53:55.319912385 (34h14m25.983861365s) [upload=35594]: Waiting for error group managing segments... Oct 31 21:53:55.332186509 (34h14m25.996135498s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.337196032 (34h14m26.001145012s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.345822268 (34h14m26.009771248s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.356624949 (34h14m26.020573918s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.357992611 (34h14m26.021941606s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.358284965 (34h14m26.022233957s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.363582356 (34h14m26.027531346s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.376206640 (34h14m26.040155641s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.386414599 (34h14m26.050363579s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.030428990 (34h15m20.694377969s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.506293399 (34h15m21.170242379s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:55.184164260 (34h15m25.848113246s) [noise=true , address=45.91.116.14:4203 , piece_id=DXLZT7TNAP7BZXQU535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection Oct 31 21:54:55.184248337 (34h15m25.848197346s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:187b9r6f2MK5c5mJEVasrhfpRHzhC4JbkWTbVvHuqB62XYs8xL, address:77.90.31.13:4641): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184319241 (34h15m25.84826822s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection Oct 31 21:54:55.184492039 (34h15m25.848441022s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184662449 (34h15m25.848611434s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: read tcp 159.223.143.39:38358->45.91.116.4:4964: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184739659 (34h15m25.848688638s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: write tcp 159.223.143.39:41986->77.90.31.14:4686: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184756157 (34h15m25.848705142s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1xwaFojWAo54xD34ZA7HACYmJf811KFqN9biJJoag5oxE9fJqx, address:45.91.116.13:4177): protocol: read tcp 159.223.143.39:41676->45.91.116.13:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184786251 (34h15m25.848735222s) [seg_pos={0 0}, upload=35594]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:54:55.206244634 (34h15m25.870193618s) [upload=35594]: Done uploading segments... Oct 31 21:54:55.206282843 (34h15m25.870231833s) [upload=35594]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 564.558908ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.....X.............................................] duration: 556.336889ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 22:02:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 556.361546ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.......X......................................X..........] duration: 556.037355ms trace: https://jaeger.dev.storj.io/trace/3adce5f38013470e FAIL (2025-10-31 22:02:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 556.114172ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X......................................X............] duration: 579.759351ms trace: https://jaeger.dev.storj.io/trace/41c2b066eb9ea5e8 FAIL (2025-10-31 22:02:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.243590254 (34h14m25.907539233s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244078634 (34h14m25.908027607s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244974253 (34h14m25.908923236s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.245239075 (34h14m25.909188054s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246191747 (34h14m25.910140732s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246721069 (34h14m25.910670051s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247101943 (34h14m25.911050925s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247367511 (34h14m25.911316501s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247624227 (34h14m25.911573229s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247887850 (34h14m25.911836838s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248392598 (34h14m25.912341583s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248921469 (34h14m25.912870457s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249532021 (34h14m25.913481008s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249926757 (34h14m25.913875738s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250408522 (34h14m25.91435751s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250881171 (34h14m25.914830153s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.251325874 (34h14m25.915274858s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.319899795 (34h14m25.983848763s) [upload=35594]: Next returned nil segment Oct 31 21:53:55.319912385 (34h14m25.983861365s) [upload=35594]: Waiting for error group managing segments... Oct 31 21:53:55.332186509 (34h14m25.996135498s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.337196032 (34h14m26.001145012s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.345822268 (34h14m26.009771248s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.356624949 (34h14m26.020573918s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.357992611 (34h14m26.021941606s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.358284965 (34h14m26.022233957s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.363582356 (34h14m26.027531346s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.376206640 (34h14m26.040155641s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.386414599 (34h14m26.050363579s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.030428990 (34h15m20.694377969s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.506293399 (34h15m21.170242379s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:55.184164260 (34h15m25.848113246s) [noise=true , address=45.91.116.14:4203 , piece_id=DXLZT7TNAP7BZXQU535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection Oct 31 21:54:55.184248337 (34h15m25.848197346s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:187b9r6f2MK5c5mJEVasrhfpRHzhC4JbkWTbVvHuqB62XYs8xL, address:77.90.31.13:4641): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184319241 (34h15m25.84826822s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection Oct 31 21:54:55.184492039 (34h15m25.848441022s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184662449 (34h15m25.848611434s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: read tcp 159.223.143.39:38358->45.91.116.4:4964: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184739659 (34h15m25.848688638s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: write tcp 159.223.143.39:41986->77.90.31.14:4686: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184756157 (34h15m25.848705142s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1xwaFojWAo54xD34ZA7HACYmJf811KFqN9biJJoag5oxE9fJqx, address:45.91.116.13:4177): protocol: read tcp 159.223.143.39:41676->45.91.116.13:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184786251 (34h15m25.848735222s) [seg_pos={0 0}, upload=35594]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:54:55.206244634 (34h15m25.870193618s) [upload=35594]: Done uploading segments... Oct 31 21:54:55.206282843 (34h15m25.870231833s) [upload=35594]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 579.785536ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..X.......X.....................................X...........] duration: 503.350167ms trace: https://jaeger.dev.storj.io/trace/49e5f4c0e8587b3d FAIL (2025-10-31 22:02:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 503.390775ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X......................................X...........] duration: 526.667799ms trace: https://jaeger.dev.storj.io/trace/41c2b066eb9ea5e8 FAIL (2025-10-31 22:02:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.243590254 (34h14m25.907539233s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244078634 (34h14m25.908027607s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244974253 (34h14m25.908923236s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.245239075 (34h14m25.909188054s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246191747 (34h14m25.910140732s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246721069 (34h14m25.910670051s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247101943 (34h14m25.911050925s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247367511 (34h14m25.911316501s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247624227 (34h14m25.911573229s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247887850 (34h14m25.911836838s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248392598 (34h14m25.912341583s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248921469 (34h14m25.912870457s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249532021 (34h14m25.913481008s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249926757 (34h14m25.913875738s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250408522 (34h14m25.91435751s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250881171 (34h14m25.914830153s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.251325874 (34h14m25.915274858s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.319899795 (34h14m25.983848763s) [upload=35594]: Next returned nil segment Oct 31 21:53:55.319912385 (34h14m25.983861365s) [upload=35594]: Waiting for error group managing segments... Oct 31 21:53:55.332186509 (34h14m25.996135498s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.337196032 (34h14m26.001145012s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.345822268 (34h14m26.009771248s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.356624949 (34h14m26.020573918s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.357992611 (34h14m26.021941606s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.358284965 (34h14m26.022233957s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.363582356 (34h14m26.027531346s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.376206640 (34h14m26.040155641s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.386414599 (34h14m26.050363579s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.030428990 (34h15m20.694377969s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.506293399 (34h15m21.170242379s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:55.184164260 (34h15m25.848113246s) [noise=true , address=45.91.116.14:4203 , piece_id=DXLZT7TNAP7BZXQU535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection Oct 31 21:54:55.184248337 (34h15m25.848197346s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:187b9r6f2MK5c5mJEVasrhfpRHzhC4JbkWTbVvHuqB62XYs8xL, address:77.90.31.13:4641): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184319241 (34h15m25.84826822s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection Oct 31 21:54:55.184492039 (34h15m25.848441022s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184662449 (34h15m25.848611434s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: read tcp 159.223.143.39:38358->45.91.116.4:4964: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184739659 (34h15m25.848688638s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: write tcp 159.223.143.39:41986->77.90.31.14:4686: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184756157 (34h15m25.848705142s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1xwaFojWAo54xD34ZA7HACYmJf811KFqN9biJJoag5oxE9fJqx, address:45.91.116.13:4177): protocol: read tcp 159.223.143.39:41676->45.91.116.13:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184786251 (34h15m25.848735222s) [seg_pos={0 0}, upload=35594]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:54:55.206244634 (34h15m25.870193618s) [upload=35594]: Done uploading segments... Oct 31 21:54:55.206282843 (34h15m25.870231833s) [upload=35594]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 526.724789ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.................................X...........] duration: 473.181659ms trace: https://jaeger.dev.storj.io/trace/236699c64467bb84 FAIL (2025-10-31 22:01:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:54.316715391 (34h14m24.980664371s) [upload=35593]: Starting upload Oct 31 21:53:54.316916021 (34h14m24.980865001s) [upload=35593]: Done starting upload Oct 31 21:53:54.316999602 (34h14m24.980948574s) [upload=35593]: Uploading segments... Oct 31 21:53:54.317521796 (34h14m24.981470776s) [upload=35593]: Got next segment. Inline: false Oct 31 21:53:54.317593157 (34h14m24.98154214s) [upload=35593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:54.317600463 (34h14m24.981549444s) [upload=35593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:54.344972306 (34h14m25.008921293s) [seg_pos={0 0}, upload=35593]: Begin upload segment... Oct 31 21:53:54.345030267 (34h14m25.008979247s) [seg_pos={0 0}, upload=35593]: Done begin upload segment. Oct 31 21:53:54.345213856 (34h14m25.009162844s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345225337 (34h14m25.009174327s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345830005 (34h14m25.009778994s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346300953 (34h14m25.010249931s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346533214 (34h14m25.010482193s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347118612 (34h14m25.011067594s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347365203 (34h14m25.011314187s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347568970 (34h14m25.01151796s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347824532 (34h14m25.01177352s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347959678 (34h14m25.011908664s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348214346 (34h14m25.012163334s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348367787 (34h14m25.012316775s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348778049 (34h14m25.012727028s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349186865 (34h14m25.013135844s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349752289 (34h14m25.013701272s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.352239055 (34h14m25.016188036s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.419826216 (34h14m25.083775196s) [upload=35593]: Next returned nil segment Oct 31 21:53:54.419839894 (34h14m25.083788873s) [upload=35593]: Waiting for error group managing segments... Oct 31 21:53:54.425324284 (34h14m25.089273265s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426679856 (34h14m25.090628835s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426719775 (34h14m25.090668754s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427269718 (34h14m25.091218697s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427363119 (34h14m25.0913121s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427761264 (34h14m25.091710245s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428214622 (34h14m25.092163602s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428234634 (34h14m25.092183614s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428729183 (34h14m25.092678166s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428832310 (34h14m25.092781292s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.429592128 (34h14m25.093541127s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.447797690 (34h14m25.111746672s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:54:24.282494056 (34h14m54.946443036s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: read tcp 159.223.143.39:35840->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282509279 (34h14m54.946458269s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 159.223.143.39:36998->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282717367 (34h14m54.946666355s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 159.223.143.39:55516->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282833181 (34h14m54.946782153s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 159.223.143.39:35952->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282879533 (34h14m54.946828499s) [seg_pos={0 0}, upload=35593]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:24.306226453 (34h14m54.970175433s) [upload=35593]: Done uploading segments... Oct 31 21:54:24.306271889 (34h14m54.970220869s) [upload=35593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 473.230622ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.......X......................................X.........] duration: 457.486165ms trace: https://jaeger.dev.storj.io/trace/3adce5f38013470e FAIL (2025-10-31 22:01:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 457.513558ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X......................................X..........] duration: 565.95884ms trace: https://jaeger.dev.storj.io/trace/41c2b066eb9ea5e8 FAIL (2025-10-31 22:01:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.243590254 (34h14m25.907539233s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244078634 (34h14m25.908027607s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244974253 (34h14m25.908923236s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.245239075 (34h14m25.909188054s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246191747 (34h14m25.910140732s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246721069 (34h14m25.910670051s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247101943 (34h14m25.911050925s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247367511 (34h14m25.911316501s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247624227 (34h14m25.911573229s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247887850 (34h14m25.911836838s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248392598 (34h14m25.912341583s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248921469 (34h14m25.912870457s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249532021 (34h14m25.913481008s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249926757 (34h14m25.913875738s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250408522 (34h14m25.91435751s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250881171 (34h14m25.914830153s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.251325874 (34h14m25.915274858s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.319899795 (34h14m25.983848763s) [upload=35594]: Next returned nil segment Oct 31 21:53:55.319912385 (34h14m25.983861365s) [upload=35594]: Waiting for error group managing segments... Oct 31 21:53:55.332186509 (34h14m25.996135498s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.337196032 (34h14m26.001145012s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.345822268 (34h14m26.009771248s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.356624949 (34h14m26.020573918s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.357992611 (34h14m26.021941606s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.358284965 (34h14m26.022233957s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.363582356 (34h14m26.027531346s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.376206640 (34h14m26.040155641s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.386414599 (34h14m26.050363579s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.030428990 (34h15m20.694377969s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.506293399 (34h15m21.170242379s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:55.184164260 (34h15m25.848113246s) [noise=true , address=45.91.116.14:4203 , piece_id=DXLZT7TNAP7BZXQU535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection Oct 31 21:54:55.184248337 (34h15m25.848197346s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:187b9r6f2MK5c5mJEVasrhfpRHzhC4JbkWTbVvHuqB62XYs8xL, address:77.90.31.13:4641): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184319241 (34h15m25.84826822s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection Oct 31 21:54:55.184492039 (34h15m25.848441022s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184662449 (34h15m25.848611434s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: read tcp 159.223.143.39:38358->45.91.116.4:4964: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184739659 (34h15m25.848688638s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: write tcp 159.223.143.39:41986->77.90.31.14:4686: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184756157 (34h15m25.848705142s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1xwaFojWAo54xD34ZA7HACYmJf811KFqN9biJJoag5oxE9fJqx, address:45.91.116.13:4177): protocol: read tcp 159.223.143.39:41676->45.91.116.13:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184786251 (34h15m25.848735222s) [seg_pos={0 0}, upload=35594]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:54:55.206244634 (34h15m25.870193618s) [upload=35594]: Done uploading segments... Oct 31 21:54:55.206282843 (34h15m25.870231833s) [upload=35594]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 565.9882ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...X.............................................X..........] duration: 603.205346ms trace: https://jaeger.dev.storj.io/trace/25b2c21daa3d085 FAIL (2025-10-31 22:01:38 UTC) orbiter warning: unexpected EOF Duration: 603.229345ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.......X......................................X........] duration: 487.514032ms trace: https://jaeger.dev.storj.io/trace/3adce5f38013470e FAIL (2025-10-31 22:01:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 487.539121ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.....X............................................] duration: 410.229282ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 22:01:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 410.256357ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...X.......X.....................................X..........] duration: 536.912282ms trace: https://jaeger.dev.storj.io/trace/49e5f4c0e8587b3d FAIL (2025-10-31 22:01:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 536.942608ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.....X...........................................] duration: 379.743972ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 22:01:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 379.792313ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X......................................X.........] duration: 466.5901ms trace: https://jaeger.dev.storj.io/trace/41c2b066eb9ea5e8 FAIL (2025-10-31 22:00:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.243590254 (34h14m25.907539233s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244078634 (34h14m25.908027607s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244974253 (34h14m25.908923236s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.245239075 (34h14m25.909188054s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246191747 (34h14m25.910140732s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246721069 (34h14m25.910670051s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247101943 (34h14m25.911050925s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247367511 (34h14m25.911316501s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247624227 (34h14m25.911573229s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247887850 (34h14m25.911836838s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248392598 (34h14m25.912341583s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248921469 (34h14m25.912870457s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249532021 (34h14m25.913481008s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249926757 (34h14m25.913875738s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250408522 (34h14m25.91435751s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250881171 (34h14m25.914830153s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.251325874 (34h14m25.915274858s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.319899795 (34h14m25.983848763s) [upload=35594]: Next returned nil segment Oct 31 21:53:55.319912385 (34h14m25.983861365s) [upload=35594]: Waiting for error group managing segments... Oct 31 21:53:55.332186509 (34h14m25.996135498s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.337196032 (34h14m26.001145012s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.345822268 (34h14m26.009771248s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.356624949 (34h14m26.020573918s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.357992611 (34h14m26.021941606s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.358284965 (34h14m26.022233957s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.363582356 (34h14m26.027531346s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.376206640 (34h14m26.040155641s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.386414599 (34h14m26.050363579s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.030428990 (34h15m20.694377969s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.506293399 (34h15m21.170242379s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:55.184164260 (34h15m25.848113246s) [noise=true , address=45.91.116.14:4203 , piece_id=DXLZT7TNAP7BZXQU535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection Oct 31 21:54:55.184248337 (34h15m25.848197346s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:187b9r6f2MK5c5mJEVasrhfpRHzhC4JbkWTbVvHuqB62XYs8xL, address:77.90.31.13:4641): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184319241 (34h15m25.84826822s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection Oct 31 21:54:55.184492039 (34h15m25.848441022s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184662449 (34h15m25.848611434s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: read tcp 159.223.143.39:38358->45.91.116.4:4964: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184739659 (34h15m25.848688638s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: write tcp 159.223.143.39:41986->77.90.31.14:4686: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184756157 (34h15m25.848705142s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1xwaFojWAo54xD34ZA7HACYmJf811KFqN9biJJoag5oxE9fJqx, address:45.91.116.13:4177): protocol: read tcp 159.223.143.39:41676->45.91.116.13:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184786251 (34h15m25.848735222s) [seg_pos={0 0}, upload=35594]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:54:55.206244634 (34h15m25.870193618s) [upload=35594]: Done uploading segments... Oct 31 21:54:55.206282843 (34h15m25.870231833s) [upload=35594]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 466.616862ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.................................X..........] duration: 492.107807ms trace: https://jaeger.dev.storj.io/trace/236699c64467bb84 FAIL (2025-10-31 22:00:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:54.316715391 (34h14m24.980664371s) [upload=35593]: Starting upload Oct 31 21:53:54.316916021 (34h14m24.980865001s) [upload=35593]: Done starting upload Oct 31 21:53:54.316999602 (34h14m24.980948574s) [upload=35593]: Uploading segments... Oct 31 21:53:54.317521796 (34h14m24.981470776s) [upload=35593]: Got next segment. Inline: false Oct 31 21:53:54.317593157 (34h14m24.98154214s) [upload=35593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:54.317600463 (34h14m24.981549444s) [upload=35593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:54.344972306 (34h14m25.008921293s) [seg_pos={0 0}, upload=35593]: Begin upload segment... Oct 31 21:53:54.345030267 (34h14m25.008979247s) [seg_pos={0 0}, upload=35593]: Done begin upload segment. Oct 31 21:53:54.345213856 (34h14m25.009162844s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345225337 (34h14m25.009174327s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345830005 (34h14m25.009778994s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346300953 (34h14m25.010249931s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346533214 (34h14m25.010482193s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347118612 (34h14m25.011067594s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347365203 (34h14m25.011314187s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347568970 (34h14m25.01151796s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347824532 (34h14m25.01177352s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347959678 (34h14m25.011908664s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348214346 (34h14m25.012163334s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348367787 (34h14m25.012316775s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348778049 (34h14m25.012727028s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349186865 (34h14m25.013135844s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349752289 (34h14m25.013701272s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.352239055 (34h14m25.016188036s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.419826216 (34h14m25.083775196s) [upload=35593]: Next returned nil segment Oct 31 21:53:54.419839894 (34h14m25.083788873s) [upload=35593]: Waiting for error group managing segments... Oct 31 21:53:54.425324284 (34h14m25.089273265s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426679856 (34h14m25.090628835s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426719775 (34h14m25.090668754s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427269718 (34h14m25.091218697s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427363119 (34h14m25.0913121s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427761264 (34h14m25.091710245s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428214622 (34h14m25.092163602s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428234634 (34h14m25.092183614s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428729183 (34h14m25.092678166s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428832310 (34h14m25.092781292s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.429592128 (34h14m25.093541127s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.447797690 (34h14m25.111746672s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:54:24.282494056 (34h14m54.946443036s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: read tcp 159.223.143.39:35840->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282509279 (34h14m54.946458269s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 159.223.143.39:36998->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282717367 (34h14m54.946666355s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 159.223.143.39:55516->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282833181 (34h14m54.946782153s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 159.223.143.39:35952->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282879533 (34h14m54.946828499s) [seg_pos={0 0}, upload=35593]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:24.306226453 (34h14m54.970175433s) [upload=35593]: Done uploading segments... Oct 31 21:54:24.306271889 (34h14m54.970220869s) [upload=35593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 492.208049ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....X.............................................X.........] duration: 634.610336ms trace: https://jaeger.dev.storj.io/trace/25b2c21daa3d085 FAIL (2025-10-31 22:00:51 UTC) orbiter warning: unexpected EOF Duration: 634.660393ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....X.......X.....................................X.........] duration: 473.301766ms trace: https://jaeger.dev.storj.io/trace/49e5f4c0e8587b3d FAIL (2025-10-31 22:00:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 473.394813ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.....X..........................................] duration: 421.636506ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 22:00:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 421.740871ms core uplink us-select-1 us1-placement-20 metainfo-projects

1
...
68
69
70
71
72
...
2975