Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 644.373909ms trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 04:59:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 644.456151ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 1.305227829s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 04:59:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.305255588s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 823.41447ms trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 04:59:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 823.441606ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X.........XXX...............................................] duration: 463.314538ms trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 04:58:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 463.391671ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 1.22541774s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 04:58:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.225446079s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.X.........XXX..............................................] duration: 521.49485ms trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 04:58:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 521.530403ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 726.655844ms trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 04:58:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 726.684984ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..X.........XXX.............................................] duration: 618.469923ms trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 04:58:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 618.516214ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 1.220432375s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 04:57:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.220463245s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [X..X.........XXX............................................] duration: 686.242155ms trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 04:57:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 686.265936ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 1.388489564s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 04:57:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.388516779s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 660.148385ms trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 04:56:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 660.183831ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.X..X.........XXX...........................................] duration: 484.549852ms trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 04:56:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 484.582716ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 695.316669ms trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 04:56:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 695.418141ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 1.374310471s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 04:56:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.374347296s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 1.292983478s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 04:55:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.293018621s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 595.37973ms trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 04:55:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 595.410155ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 1.378549776s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 04:55:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.378584427s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..X..X.........XXX..........................................] duration: 519.870272ms trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 04:55:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 520.090561ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...X..X.........XXX.........................................] duration: 761.248236ms trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 04:54:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 761.279171ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 701.56557ms trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 04:54:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 701.595108ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 1.515582121s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 04:54:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.515644516s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [....X..X.........XXX........................................] duration: 712.347303ms trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 04:54:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 712.43357ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 1.273745348s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 04:54:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.273772499s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 556.068606ms trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 04:53:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 556.107273ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 1.751291075s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 04:53:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.75131435s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.....X..X.........XXX.......................................] duration: 533.079971ms trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 04:53:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 533.121028ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 723.012353ms trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 04:52:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 723.040187ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 1.286427457s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 04:52:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.286524719s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 725.624966ms trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 04:52:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 725.734962ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [......X..X.........XXX......................................] duration: 470.733634ms trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 04:51:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 470.765683ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 1.381409992s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 04:51:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.381527673s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 617.216184ms trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 04:51:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 617.257388ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.......X..X.........XXX.....................................] duration: 515.256411ms trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 04:51:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 515.284797ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 1.365857061s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 04:50:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.365892077s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 670.262122ms trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 04:50:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 670.356825ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 1.418258239s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 04:50:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.418290194s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [........X..X.........XXX....................................] duration: 452.262204ms trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 04:49:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 452.278919ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 532.030117ms trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 04:49:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 532.086375ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 1.339301376s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 04:49:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.339327943s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 641.913007ms trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 04:49:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 641.94884ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.........X..X.........XXX...................................] duration: 447.565711ms trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 04:48:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 447.624588ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.369548027s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 04:48:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.369578318s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 576.837702ms trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 04:48:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 576.900664ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..........X..X.........XXX..................................] duration: 470.347382ms trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 04:48:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 470.396406ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...........X..X.........XXX.................................] duration: 433.940688ms trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 04:47:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 434.02299ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.300586096s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 04:47:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.300651398s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.563830595s trace: https://jaeger.dev.storj.io/trace/5cce09de575b8506 FAIL (2026-04-04 04:47:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6881916s) [upload=772064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 4 04:21:44.460130782 (689h21m42.056901815s) [upload=772064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 4 04:21:44.500551580 (689h21m42.097322604s) [seg_pos={0 0}, upload=772064]: Begin upload segment... Apr 4 04:21:44.500628241 (689h21m42.097399263s) [seg_pos={0 0}, upload=772064]: Done begin upload segment. Apr 4 04:21:44.500668892 (689h21m42.097439922s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.500792978 (689h21m42.097564014s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501687053 (689h21m42.09845809s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.501828492 (689h21m42.098599515s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502343164 (689h21m42.099114198s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.502900735 (689h21m42.099671764s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503495842 (689h21m42.100266868s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.503796813 (689h21m42.100567839s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504006258 (689h21m42.100777287s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504542000 (689h21m42.101313035s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.504560079 (689h21m42.101331115s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505047220 (689h21m42.101818243s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505183875 (689h21m42.10195491s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.505619570 (689h21m42.102390603s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506074686 (689h21m42.102845722s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.506877697 (689h21m42.103648727s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507462874 (689h21m42.104233903s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.507975323 (689h21m42.10474639s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Uploading piece... Apr 4 04:21:44.582999887 (689h21m42.17977091s) [upload=772064]: Next returned nil segment Apr 4 04:21:44.583065715 (689h21m42.179836738s) [upload=772064]: Waiting for error group managing segments... Apr 4 04:21:44.587179313 (689h21m42.183950339s) [noise=true , address=45.91.116.12:4229 , piece_id=CDXY4QA2BKVNNJUHGL2JPK3MMHTVDPRN64SAQSMZSVQIG755ZOJQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.587315638 (689h21m42.184086655s) [noise=true , address=45.91.116.13:4158 , piece_id=OX4BJEH3DNMAUXS64QHDHRK75YYEF6DPQJAX7MS3J6K7RUM4GM2Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.592227274 (689h21m42.188998305s) [noise=true , address=145.79.227.8:4345 , piece_id=2UWIKM3L3JUKNYWPBVJBEGFOEWS3WVM24AY7C2OVBZWDAXDG4I3A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:44.897574490 (689h21m42.494345523s) [noise=true , address=167.17.72.70:4751 , piece_id=AX7C5IEHMVNPGGAB7WPWUOSFEHVWWA75JGDAH3DAGZVZCVMEZKKQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084711646 (689h21m42.681482669s) [noise=true , address=193.149.164.66:4996 , piece_id=YGOQ6INTPJ3AZW3RSTVH2IWITNFONA7JTW25BIXNBZRRZUN6YRYA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.084884121 (689h21m42.681655134s) [noise=true , address=193.149.164.67:4002 , piece_id=CYJYJ5ZNXT63OT5DBDLW7ML32OIZLGNWMLM4W2F5TPSSJ2QJ2NBA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.089152844 (689h21m42.685923867s) [noise=true , address=193.149.164.70:4115 , piece_id=EHMHJHJ2ZQWAJWBFSHOTAXWE7QJQOIICXDKAK33KMOAF7C7KZ2PA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.103385369 (689h21m42.700156382s) [noise=true , address=193.149.164.68:4062 , piece_id=IM6B4U2ARVUSBX4276OSQPEPQNMWMFSUUOYAEJ3ZG2GLFUKGGQGQ, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.127711185 (689h21m42.724482213s) [noise=true , address=193.149.164.69:4084 , piece_id=BUV3LZ7PDHSGRNKHJSSN2OFQYMIYUELWEPO4PF3DMTBX65S4YPMA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.143513369 (689h21m42.740284381s) [noise=true , address=193.149.164.71:4308 , piece_id=YGF5F7J25BEULKSTR3ZDC3HE4EHVJ3IF4HQCNATVWHITGRB344JA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.307124646 (689h21m42.903895674s) [noise=true , address=167.17.72.68:4656 , piece_id=YFAXDPM4TYBLYL4UR3Z6FDCDJXU34J7XRURDGLXZAMUOYYZL56IA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:45.517599945 (689h21m43.114370966s) [noise=true , address=145.79.227.33:4898 , piece_id=64X2NB2BPWLP4C5EC53A6EKFPNQCJI6BGEDY6BFE2RFNYEPCDA5Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:21:46.061904485 (689h21m43.658675509s) [noise=true , address=193.149.164.143:4704 , piece_id=PZ2W2J22M5MZIYBFEOJCEOIVAJOJGYCHCQUPDSELS764HZG4OF2A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: <nil> Apr 4 04:22:44.423237599 (689h22m42.020008631s) [noise=true , address=77.90.31.13:4658 , piece_id=I7GGGCPJIHSQ5RYYOPIXKWAJBOMSRSK3ZDCASMKUPBQPALOJCKXA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:50834->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423364101 (689h22m42.020135144s) [noise=true , address=77.90.31.17:4783 , piece_id=E2XPGXQAVSYEMBWOIPKHGL5MVVGC2FSLN5KHZQC6DBUAT3VUWOKA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: read tcp 159.223.143.39:45990->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423435607 (689h22m42.020206643s) [noise=true , address=77.90.31.16:4750 , piece_id=TI2D4Z2MYSXWBHX3ZVEMRFLI4LW5RDVCENH43S7GEFR7WGIVCI6A, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 159.223.143.39:34564->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423560856 (689h22m42.020332015s) [noise=true , address=77.90.31.18:4823 , piece_id=MI45FJCGXJ2CIL4OOVHEUB7TJXEGH6WELCSTKR6JD753YLF5UN4Q, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: read tcp 159.223.143.39:40028->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423793037 (689h22m42.020564071s) [noise=true , address=77.90.31.15:4709 , piece_id=HQLXOXRSN7R5WC5K2W7X4YUBVEZFITOY4VEYKG7J6WIJDEHABWGA, seg_pos={0 0}, upload=772064]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: read tcp 159.223.143.39:59234->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 4 04:22:44.423864095 (689h22m42.020635106s) [seg_pos={0 0}, upload=772064]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 4 04:22:44.451342916 (689h22m42.048113939s) [upload=772064]: Done uploading segments... Apr 4 04:22:44.451381831 (689h22m42.048152855s) [upload=772064]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.563971333s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 626.585024ms trace: https://jaeger.dev.storj.io/trace/670ee369f5f2d334 FAIL (2026-04-04 04:47:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 626.63609ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [............X..X.........XXX................................] duration: 450.147252ms trace: https://jaeger.dev.storj.io/trace/3304a822f59853ad FAIL (2026-04-04 04:47:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 450.24607ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)