Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 313265 reports (Page 32 of 6266)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.64505771s trace: https://jaeger.dev.storj.io/trace/600cf5755696a8b6 FAIL (2026-04-04 07:44:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.645158951s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.57190604s trace: https://jaeger.dev.storj.io/trace/600cf5755696a8b6 FAIL (2026-04-04 07:43:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.571941795s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.545444674s trace: https://jaeger.dev.storj.io/trace/600cf5755696a8b6 FAIL (2026-04-04 07:43:14 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.54548102s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.638226394s trace: https://jaeger.dev.storj.io/trace/600cf5755696a8b6 FAIL (2026-04-04 07:42:27 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.638293606s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.527540765s trace: https://jaeger.dev.storj.io/trace/600cf5755696a8b6 FAIL (2026-04-04 07:41:58 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.527579221s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.646490696s trace: https://jaeger.dev.storj.io/trace/600cf5755696a8b6 FAIL (2026-04-04 07:40:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.646519724s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.603662268s trace: https://jaeger.dev.storj.io/trace/600cf5755696a8b6 FAIL (2026-04-04 07:40:11 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.603765148s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.711143349s trace: https://jaeger.dev.storj.io/trace/600cf5755696a8b6 FAIL (2026-04-04 07:39:44 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.711175427s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.543998391s trace: https://jaeger.dev.storj.io/trace/600cf5755696a8b6 FAIL (2026-04-04 07:38:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.544033205s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.550982898s trace: https://jaeger.dev.storj.io/trace/600cf5755696a8b6 FAIL (2026-04-04 07:38:11 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.551027225s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.740540926s trace: https://jaeger.dev.storj.io/trace/600cf5755696a8b6 FAIL (2026-04-04 07:37:14 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.740662281s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.668604046s trace: https://jaeger.dev.storj.io/trace/600cf5755696a8b6 FAIL (2026-04-04 07:36:35 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.668636231s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.666081402s trace: https://jaeger.dev.storj.io/trace/600cf5755696a8b6 FAIL (2026-04-04 07:35:41 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.666120578s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 50.925692341s trace: https://jaeger.dev.storj.io/trace/600cf5755696a8b6 FAIL (2026-04-04 07:35:16 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 50.925742086s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 1.300478231s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 05:09:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.30069592s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 500.277326ms trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 05:08:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 500.33016ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 1.333354062s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 05:08:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.333449689s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [XX..........................................................] duration: 474.433376ms trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 05:07:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 474.467001ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 658.889862ms trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 05:07:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 658.927647ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [XXX.........................................................] duration: 656.482407ms trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 05:07:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 656.525721ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 1.47994359s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 05:07:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.480024159s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 594.007057ms trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 05:06:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 594.044131ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 644.454415ms trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 05:06:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 644.50721ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 1.226511563s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 05:06:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.226547736s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.XXX........................................................] duration: 751.133318ms trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 05:06:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 751.161298ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 767.847299ms trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 05:05:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 768.001974ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 576.771989ms trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 05:05:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 576.816377ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 1.418629028s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 05:05:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.418672062s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 576.533851ms trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 05:04:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 576.650567ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..XXX.......................................................] duration: 603.035255ms trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 05:04:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 603.076735ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 650.447397ms trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 05:04:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 650.586743ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 1.491579674s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 05:04:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.491678881s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 1.458463754s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 05:03:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.458578312s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...XXX......................................................] duration: 1.002841279s trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 05:03:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.002870811s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 974.930529ms trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 05:03:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 974.964159ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 1.384555583s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 05:03:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.384576483s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....XXX.....................................................] duration: 634.927502ms trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 05:02:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 634.963397ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 804.025893ms trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 05:02:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 804.09128ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 1.365379635s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 05:02:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.365483933s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....XXX....................................................] duration: 917.490331ms trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 05:02:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 917.517333ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......XXX...................................................] duration: 1.161347068s trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 05:02:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.161386033s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 1.543621214s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 05:02:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.543653679s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 1.190996326s trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 05:02:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.191116469s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 1.452498026s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 05:01:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.45252436s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......XXX..................................................] duration: 1.01039928s trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 05:01:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.010436323s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 630.20483ms trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 05:01:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 630.279904ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........XXX.................................................] duration: 536.549434ms trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 05:00:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 536.646624ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 1.064548752s trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 05:00:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.064618739s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 1.434115452s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 05:00:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.434146922s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........XXX................................................] duration: 515.357897ms trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 04:59:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 515.530423ms core uplink us-select-1 us1-placement-20

1
...
30
31
32
33
34
...
6266