Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
17 out of 60 is failed (threshold=3) [..........................................XXXXXXX.XXXXXXXXXX] duration:
1m0.026994385s trace:
https://jaeger.dev.storj.io/trace/6f1bd04ac60ad90c
FAIL (2025-12-10 23:41:30 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
31.753310613s) [upload=133318]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:40:30.954979711 (128h12m31.782688106s) [seg_pos={0 0}, upload=133318]: Begin upload segment...
Dec 10 23:40:30.955199046 (128h12m31.782907438s) [seg_pos={0 0}, upload=133318]: Done begin upload segment.
Dec 10 23:40:30.955230849 (128h12m31.782939249s) [noise=true , address=145.79.227.50:4398 , piece_id=HJ5HC36HRTOKWZDTUMZZSSJQJFNQAS5F6CKZ7AN5M5INVOKE6KVQ, seg_pos={0 0}, upload=133318]: Uploading piece...
Dec 10 23:40:30.955264154 (128h12m31.782972545s) [noise=true , address=77.90.31.21:4902 , piece_id=BH7PSIZUG66ZN22CENIUGTEYXKYUCE3YL2K44NENAP5X7FNTGEYA, seg_pos={0 0}, upload=133318]: Uploading piece...
Dec 10 23:40:30.955355227 (128h12m31.783063617s) [noise=true , address=45.91.116.7:4020 , piece_id=U5E3S5YUYTQYRRL3OTRZC7EWNQOGHTXWDCRLXQOFXMZT5PNVEXCQ, seg_pos={0 0}, upload=133318]: Uploading piece...
Dec 10 23:40:30.955936988 (128h12m31.783645384s) [noise=true , address=77.90.31.18:4836 , piece_id=UDVCLGDLU6FK5U27CHXBBGMV4BP4V463GHSMRRY4B5WIT46G2FJA, seg_pos={0 0}, upload=133318]: Uploading piece...
Dec 10 23:40:30.956389961 (128h12m31.784098359s) [noise=true , address=145.79.227.19:4093 , piece_id=3GTCS3S5Z7ZD7CAD2GO5DZ5GEY4PMT22ZGR2A2NNRNMX7MGK4U5Q, seg_pos={0 0}, upload=133318]: Uploading piece...
Dec 10 23:40:30.956696510 (128h12m31.78440492s) [noise=true , address=45.91.116.8:4055 , piece_id=EK2KYG5T6LPQ5LVIKI6ELUR5OYQK5QZ7GV6OKBHS3GHZOI36D4DQ, seg_pos={0 0}, upload=133318]: Uploading piece...
Dec 10 23:40:30.957306124 (128h12m31.785014515s) [noise=true , address=145.79.227.52:4466 , piece_id=5M5JXG474W5QSAT4TWXGC6UWVB6IA6HIUJXRTSEABZKZCW4KN6GQ, seg_pos={0 0}, upload=133318]: Uploading piece...
Dec 10 23:40:30.960719429 (128h12m31.788427824s) [noise=true , address=45.91.116.13:4178 , piece_id=KWQ7PXK4TQXU5JAHN3G3E3FXORVEJBZH5AQM4BU2HIN2IHOB36JQ, seg_pos={0 0}, upload=133318]: Uploading piece...
Dec 10 23:40:30.961352134 (128h12m31.789060531s) [noise=true , address=77.90.31.11:4583 , piece_id=6J3YZVGSELLCOGS5FAEX62E3RF3OTAOTPWDVGJ2RHOHSD2QVQ53A, seg_pos={0 0}, upload=133318]: Uploading piece...
Dec 10 23:40:30.961890294 (128h12m31.789598695s) [noise=true , address=77.90.31.13:4656 , piece_id=VWCVGDQ4XL5JEYJ64ABSBQ74YJ52JCTBDDE4SOYUTZNEWEH6SYBQ, seg_pos={0 0}, upload=133318]: Uploading piece...
Dec 10 23:40:30.962366612 (128h12m31.790075009s) [noise=true , address=45.91.116.12:4232 , piece_id=YLTVG6SS662MHWQUMNUAUKD4RGLHMJDCGN54E4XKRCTDB3JTGPAQ, seg_pos={0 0}, upload=133318]: Uploading piece...
Dec 10 23:40:30.962852462 (128h12m31.790560859s) [noise=true , address=5.231.73.3:4018 , piece_id=CE2ZTN4OLWPMPFMFMA2J4QSU4O7TNZPL346LEUKHPUZYSBBOCR3A, seg_pos={0 0}, upload=133318]: Uploading piece...
Dec 10 23:40:30.964801756 (128h12m31.79251015s) [noise=true , address=45.91.116.6:4997 , piece_id=G6ORWTLU5ELHPIIOZHNIDVEFFNX67EAGC67UEYJ5ANKTEQ3S2NJQ, seg_pos={0 0}, upload=133318]: Uploading piece...
Dec 10 23:40:30.965324770 (128h12m31.793033166s) [noise=true , address=45.91.116.4:4967 , piece_id=A4RGZLN5TRV5UJYNWP67UAQYJZ63SFCAK4LCJWLGTZY536KFIY7Q, seg_pos={0 0}, upload=133318]: Uploading piece...
Dec 10 23:40:30.967622810 (128h12m31.795331213s) [noise=true , address=145.79.227.30:4786 , piece_id=B7YMWU2XTHCGLQSEAXG3HINIIERQ35V56AOYMEQK3GHG2KDWOGZA, seg_pos={0 0}, upload=133318]: Uploading piece...
Dec 10 23:40:30.968358720 (128h12m31.796067122s) [noise=true , address=77.90.31.12:4605 , piece_id=7ASQW7KTOYAK2LXOC7PRHXECJ22KH4L5JCEEIIBTMMUHQ6BFLADA, seg_pos={0 0}, upload=133318]: Uploading piece...
Dec 10 23:40:30.969078733 (128h12m31.796787131s) [noise=true , address=77.90.31.10:4537 , piece_id=FG2MKLFDIDEAU3A5RZ5IQVGZARSO5EX3JFCTNCQRHEXYGWJ3GOVQ, seg_pos={0 0}, upload=133318]: Uploading piece...
Dec 10 23:40:30.969711508 (128h12m31.797419909s) [noise=true , address=145.79.227.33:4892 , piece_id=AN4HMGGQLAQC3MKJLTQKXWUTT56OSR44O5B7ZGAIKL7BCFNKQJXA, seg_pos={0 0}, upload=133318]: Uploading piece...
Dec 10 23:40:31.045147032 (128h12m31.872855423s) [upload=133318]: Next returned nil segment
Dec 10 23:40:31.045160575 (128h12m31.872868967s) [upload=133318]: Waiting for error group managing segments...
Dec 10 23:40:31.053308978 (128h12m31.881017369s) [noise=true , address=45.91.116.6:4997 , piece_id=G6ORWTLU5ELHPIIOZHNIDVEFFNX67EAGC67UEYJ5ANKTEQ3S2NJQ, seg_pos={0 0}, upload=133318]: Done uploading piece. err: <nil>
Dec 10 23:40:31.055332344 (128h12m31.883040743s) [noise=true , address=45.91.116.12:4232 , piece_id=YLTVG6SS662MHWQUMNUAUKD4RGLHMJDCGN54E4XKRCTDB3JTGPAQ, seg_pos={0 0}, upload=133318]: Done uploading piece. err: <nil>
Dec 10 23:40:31.058103966 (128h12m31.885812388s) [noise=true , address=145.79.227.50:4398 , piece_id=HJ5HC36HRTOKWZDTUMZZSSJQJFNQAS5F6CKZ7AN5M5INVOKE6KVQ, seg_pos={0 0}, upload=133318]: Done uploading piece. err: <nil>
Dec 10 23:40:31.062416826 (128h12m31.89012522s) [noise=true , address=145.79.227.33:4892 , piece_id=AN4HMGGQLAQC3MKJLTQKXWUTT56OSR44O5B7ZGAIKL7BCFNKQJXA, seg_pos={0 0}, upload=133318]: Done uploading piece. err: <nil>
Dec 10 23:40:31.064063731 (128h12m31.891772132s) [noise=true , address=45.91.116.4:4967 , piece_id=A4RGZLN5TRV5UJYNWP67UAQYJZ63SFCAK4LCJWLGTZY536KFIY7Q, seg_pos={0 0}, upload=133318]: Done uploading piece. err: <nil>
Dec 10 23:40:31.065397440 (128h12m31.893105831s) [noise=true , address=45.91.116.13:4178 , piece_id=KWQ7PXK4TQXU5JAHN3G3E3FXORVEJBZH5AQM4BU2HIN2IHOB36JQ, seg_pos={0 0}, upload=133318]: Done uploading piece. err: <nil>
Dec 10 23:40:31.073058297 (128h12m31.900766691s) [noise=true , address=45.91.116.7:4020 , piece_id=U5E3S5YUYTQYRRL3OTRZC7EWNQOGHTXWDCRLXQOFXMZT5PNVEXCQ, seg_pos={0 0}, upload=133318]: Done uploading piece. err: <nil>
Dec 10 23:40:31.073119297 (128h12m31.900827692s) [noise=true , address=45.91.116.8:4055 , piece_id=EK2KYG5T6LPQ5LVIKI6ELUR5OYQK5QZ7GV6OKBHS3GHZOI36D4DQ, seg_pos={0 0}, upload=133318]: Done uploading piece. err: <nil>
Dec 10 23:40:31.073539467 (128h12m31.901247861s) [noise=true , address=145.79.227.19:4093 , piece_id=3GTCS3S5Z7ZD7CAD2GO5DZ5GEY4PMT22ZGR2A2NNRNMX7MGK4U5Q, seg_pos={0 0}, upload=133318]: Done uploading piece. err: <nil>
Dec 10 23:40:31.101210913 (128h12m31.928919303s) [noise=true , address=145.79.227.52:4466 , piece_id=5M5JXG474W5QSAT4TWXGC6UWVB6IA6HIUJXRTSEABZKZCW4KN6GQ, seg_pos={0 0}, upload=133318]: Done uploading piece. err: <nil>
Dec 10 23:40:31.157571930 (128h12m31.985280325s) [noise=true , address=5.231.73.3:4018 , piece_id=CE2ZTN4OLWPMPFMFMA2J4QSU4O7TNZPL346LEUKHPUZYSBBOCR3A, seg_pos={0 0}, upload=133318]: Done uploading piece. err: <nil>
Dec 10 23:40:32.015318350 (128h12m32.843026744s) [noise=true , address=145.79.227.30:4786 , piece_id=B7YMWU2XTHCGLQSEAXG3HINIIERQ35V56AOYMEQK3GHG2KDWOGZA, seg_pos={0 0}, upload=133318]: Done uploading piece. err: <nil>
Dec 10 23:41:30.884965851 (128h13m31.712674254s) [noise=true , address=77.90.31.12:4605 , piece_id=7ASQW7KTOYAK2LXOC7PRHXECJ22KH4L5JCEEIIBTMMUHQ6BFLADA, seg_pos={0 0}, upload=133318]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: rpc: dial tcp 77.90.31.12:4605: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:41:30.885247713 (128h13m31.712956115s) [noise=true , address=77.90.31.11:4583 , piece_id=6J3YZVGSELLCOGS5FAEX62E3RF3OTAOTPWDVGJ2RHOHSD2QVQ53A, seg_pos={0 0}, upload=133318]: Done uploading piece. err: ecclient: upload failed (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t, address:77.90.31.11:4583): protocol: rpc: dial tcp 77.90.31.11:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:41:30.885260607 (128h13m31.712969007s) [noise=true , address=77.90.31.10:4537 , piece_id=FG2MKLFDIDEAU3A5RZ5IQVGZARSO5EX3JFCTNCQRHEXYGWJ3GOVQ, seg_pos={0 0}, upload=133318]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: rpc: dial tcp 77.90.31.10:4537: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:41:30.885433154 (128h13m31.713141574s) [noise=true , address=77.90.31.13:4656 , piece_id=VWCVGDQ4XL5JEYJ64ABSBQ74YJ52JCTBDDE4SOYUTZNEWEH6SYBQ, seg_pos={0 0}, upload=133318]: Done uploading piece. err: ecclient: upload failed (node:12CEXqfegKNoaZ516cRAYvaMBygJVdMtvyrArc85zK3eDQC6tXN, address:77.90.31.13:4656): protocol: write tcp 159.223.143.39:51604->77.90.31.13:4656: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:41:30.885574806 (128h13m31.713283206s) [noise=true , address=77.90.31.18:4836 , piece_id=UDVCLGDLU6FK5U27CHXBBGMV4BP4V463GHSMRRY4B5WIT46G2FJA, seg_pos={0 0}, upload=133318]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:41:30.885820476 (128h13m31.71352887s) [noise=true , address=77.90.31.21:4902 , piece_id=BH7PSIZUG66ZN22CENIUGTEYXKYUCE3YL2K44NENAP5X7FNTGEYA, seg_pos={0 0}, upload=133318]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 159.223.143.39:59948->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:41:30.885868677 (128h13m31.713577069s) [seg_pos={0 0}, upload=133318]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:41:30.909078398 (128h13m31.736786789s) [upload=133318]: Done uploading segments...
Dec 10 23:41:30.909117022 (128h13m31.736825416s) [upload=133318]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1m0.027011617s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...................................X....X...X........X...X..] duration:
450.947429ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:41:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 450.981011ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [....................................XXX.....................] duration:
534.145078ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:41:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 534.302255ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.....................................XXX....................] duration:
570.770506ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:40:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 570.80232ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [....................................XXX.X...................] duration:
460.684391ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:40:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 460.707712ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
16 out of 60 is failed (threshold=3) [...........................................XXXXXXX.XXXXXXXXX] duration:
1m0.026165123s trace:
https://jaeger.dev.storj.io/trace/f69676ee8fdfd7d
FAIL (2025-12-10 23:40:28 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:39:28.262488102 (128h11m29.0901965s) [upload=133302]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:39:28.301215424 (128h11m29.128923827s) [seg_pos={0 0}, upload=133302]: Begin upload segment...
Dec 10 23:39:28.301306710 (128h11m29.129015105s) [seg_pos={0 0}, upload=133302]: Done begin upload segment.
Dec 10 23:39:28.301318882 (128h11m29.129027283s) [noise=true , address=77.90.31.21:4896 , piece_id=HSU2N276S5IBLYZ2XBLS6X5LR4ETV5HVS6BPKEOKEE4FHB3H4BQA, seg_pos={0 0}, upload=133302]: Uploading piece...
Dec 10 23:39:28.301359039 (128h11m29.12906744s) [noise=true , address=145.79.227.12:4797 , piece_id=4FC6ZTCGJOG2VGQGLW4KPONQIIVSJ2N6BAPJM6GNBWGLNQZPAWRQ, seg_pos={0 0}, upload=133302]: Uploading piece...
Dec 10 23:39:28.301926653 (128h11m29.129635053s) [noise=true , address=77.90.31.19:4870 , piece_id=IIWEVQVA5HWFVQOQK2SCENPB6RMYF2HUBKWVOGQA6N5RUFBDI2UA, seg_pos={0 0}, upload=133302]: Uploading piece...
Dec 10 23:39:28.302079808 (128h11m29.129788203s) [noise=true , address=77.90.31.17:4787 , piece_id=2APBQIBNWIWGHZYAIYNDGMDTNXGJWRU6JELGAQCSTW3FDHHXQLJQ, seg_pos={0 0}, upload=133302]: Uploading piece...
Dec 10 23:39:28.302995341 (128h11m29.130703745s) [noise=true , address=145.79.227.29:4962 , piece_id=WKTTUDHE7ZNXBVRWYGI4HVRDZAQH6RH7P5SLUILFGNND4KC3KY7Q, seg_pos={0 0}, upload=133302]: Uploading piece...
Dec 10 23:39:28.303481464 (128h11m29.131189866s) [noise=true , address=5.231.73.4:4035 , piece_id=TAE6K7WKY7465O6YFIBSAO43JYNZ2ZVL5CBONAEIDX36DO4QVM7A, seg_pos={0 0}, upload=133302]: Uploading piece...
Dec 10 23:39:28.304408058 (128h11m29.132116458s) [noise=true , address=77.90.31.13:4658 , piece_id=T6VI4T5UGDK67KY6URNU2G4RZPRAPQ2NRHPYBZNCXPFF7O5FIF3Q, seg_pos={0 0}, upload=133302]: Uploading piece...
Dec 10 23:39:28.304985969 (128h11m29.132694365s) [noise=true , address=45.91.116.5:4327 , piece_id=AWTETCWG5DMEJX4VL6Q2GRQABIN5CUYX7H36PFIZBAIM3CRBPYBA, seg_pos={0 0}, upload=133302]: Uploading piece...
Dec 10 23:39:28.305491866 (128h11m29.133200263s) [noise=true , address=77.90.31.16:4738 , piece_id=2K2L3SRXBCLEFLJ45CZVN7UMYWIE7V7RJSAYSCP5KHJHAU4IN42Q, seg_pos={0 0}, upload=133302]: Uploading piece...
Dec 10 23:39:28.305913093 (128h11m29.133621486s) [noise=true , address=45.91.116.9:4271 , piece_id=R57IDREWZIA5PLN7VJK33GE4AALCABEYLPQMHWFDBHYZTSIFOUXA, seg_pos={0 0}, upload=133302]: Uploading piece...
Dec 10 23:39:28.306273820 (128h11m29.133982211s) [noise=true , address=45.91.116.12:4252 , piece_id=L36VLBEK77ZOTGD6YLT5QWZUMTXLGVT5XT2HHOMRHDROOXTCA2VA, seg_pos={0 0}, upload=133302]: Uploading piece...
Dec 10 23:39:28.307147428 (128h11m29.134855828s) [noise=true , address=145.79.227.19:4099 , piece_id=T4NOMA37NQD73JEWVZE26SANXFYK7JXEYX4OTUPGCVIODY2VFMZA, seg_pos={0 0}, upload=133302]: Uploading piece...
Dec 10 23:39:28.307657324 (128h11m29.135365725s) [noise=true , address=64.49.13.140:4509 , piece_id=6E7FHYYLZZAFOJST74ZQ6CSNFI2JLV3TC2GR6E6MCIUV6B5KKMUQ, seg_pos={0 0}, upload=133302]: Uploading piece...
Dec 10 23:39:28.308236847 (128h11m29.135945229s) [noise=true , address=45.91.116.14:4220 , piece_id=Y5KIFKG4HZ7FGNICH23YYSCSRVK4HRKP3FYBYXIBY3W4CPFSG4XQ, seg_pos={0 0}, upload=133302]: Uploading piece...
Dec 10 23:39:28.309754396 (128h11m29.137462793s) [noise=true , address=145.79.227.35:4505 , piece_id=HXBRPB7TBNBEF2JOD75XDBM6JIGLE4V7ZH4MK2VI2BVDJMX7HGOQ, seg_pos={0 0}, upload=133302]: Uploading piece...
Dec 10 23:39:28.310375438 (128h11m29.138083836s) [noise=true , address=45.91.116.13:4160 , piece_id=Z2ZISHER52V5NCDWN7IXQDA33YFI6XEPJ7DPQSWGHH47XRBOTR5A, seg_pos={0 0}, upload=133302]: Uploading piece...
Dec 10 23:39:28.310966679 (128h11m29.138675073s) [noise=true , address=77.90.31.12:4603 , piece_id=QZHKCKRHMKFECKN3JK6XBSMHYT75PYDTYHP5LRPVNSSI5TTIWUDQ, seg_pos={0 0}, upload=133302]: Uploading piece...
Dec 10 23:39:28.311777695 (128h11m29.139486095s) [noise=true , address=145.79.227.30:4774 , piece_id=2EQHK22B3AQEQEOAN24OUUOGVKCRSXHPRONJVSXLRBARERSBT7CQ, seg_pos={0 0}, upload=133302]: Uploading piece...
Dec 10 23:39:28.374597291 (128h11m29.202305685s) [upload=133302]: Next returned nil segment
Dec 10 23:39:28.374617388 (128h11m29.202325782s) [upload=133302]: Waiting for error group managing segments...
Dec 10 23:39:28.379756801 (128h11m29.207465182s) [noise=true , address=45.91.116.13:4160 , piece_id=Z2ZISHER52V5NCDWN7IXQDA33YFI6XEPJ7DPQSWGHH47XRBOTR5A, seg_pos={0 0}, upload=133302]: Done uploading piece. err: <nil>
Dec 10 23:39:28.381678343 (128h11m29.209386733s) [noise=true , address=45.91.116.9:4271 , piece_id=R57IDREWZIA5PLN7VJK33GE4AALCABEYLPQMHWFDBHYZTSIFOUXA, seg_pos={0 0}, upload=133302]: Done uploading piece. err: <nil>
Dec 10 23:39:28.391691978 (128h11m29.219400371s) [noise=true , address=45.91.116.5:4327 , piece_id=AWTETCWG5DMEJX4VL6Q2GRQABIN5CUYX7H36PFIZBAIM3CRBPYBA, seg_pos={0 0}, upload=133302]: Done uploading piece. err: <nil>
Dec 10 23:39:28.391762424 (128h11m29.219470802s) [noise=true , address=145.79.227.19:4099 , piece_id=T4NOMA37NQD73JEWVZE26SANXFYK7JXEYX4OTUPGCVIODY2VFMZA, seg_pos={0 0}, upload=133302]: Done uploading piece. err: <nil>
Dec 10 23:39:28.393517709 (128h11m29.221226103s) [noise=true , address=45.91.116.14:4220 , piece_id=Y5KIFKG4HZ7FGNICH23YYSCSRVK4HRKP3FYBYXIBY3W4CPFSG4XQ, seg_pos={0 0}, upload=133302]: Done uploading piece. err: <nil>
Dec 10 23:39:28.397508264 (128h11m29.225216655s) [noise=true , address=145.79.227.35:4505 , piece_id=HXBRPB7TBNBEF2JOD75XDBM6JIGLE4V7ZH4MK2VI2BVDJMX7HGOQ, seg_pos={0 0}, upload=133302]: Done uploading piece. err: <nil>
Dec 10 23:39:28.397789440 (128h11m29.225497831s) [noise=true , address=45.91.116.12:4252 , piece_id=L36VLBEK77ZOTGD6YLT5QWZUMTXLGVT5XT2HHOMRHDROOXTCA2VA, seg_pos={0 0}, upload=133302]: Done uploading piece. err: <nil>
Dec 10 23:39:28.409185400 (128h11m29.236893794s) [noise=true , address=145.79.227.12:4797 , piece_id=4FC6ZTCGJOG2VGQGLW4KPONQIIVSJ2N6BAPJM6GNBWGLNQZPAWRQ, seg_pos={0 0}, upload=133302]: Done uploading piece. err: <nil>
Dec 10 23:39:28.466823553 (128h11m29.294531931s) [noise=true , address=5.231.73.4:4035 , piece_id=TAE6K7WKY7465O6YFIBSAO43JYNZ2ZVL5CBONAEIDX36DO4QVM7A, seg_pos={0 0}, upload=133302]: Done uploading piece. err: <nil>
Dec 10 23:39:28.597292088 (128h11m29.425000483s) [noise=true , address=145.79.227.29:4962 , piece_id=WKTTUDHE7ZNXBVRWYGI4HVRDZAQH6RH7P5SLUILFGNND4KC3KY7Q, seg_pos={0 0}, upload=133302]: Done uploading piece. err: <nil>
Dec 10 23:39:28.901926811 (128h11m29.729635211s) [noise=true , address=64.49.13.140:4509 , piece_id=6E7FHYYLZZAFOJST74ZQ6CSNFI2JLV3TC2GR6E6MCIUV6B5KKMUQ, seg_pos={0 0}, upload=133302]: Done uploading piece. err: <nil>
Dec 10 23:39:29.334881504 (128h11m30.162589896s) [noise=true , address=145.79.227.30:4774 , piece_id=2EQHK22B3AQEQEOAN24OUUOGVKCRSXHPRONJVSXLRBARERSBT7CQ, seg_pos={0 0}, upload=133302]: Done uploading piece. err: <nil>
Dec 10 23:40:28.216913347 (128h12m29.044621747s) [noise=true , address=77.90.31.19:4870 , piece_id=IIWEVQVA5HWFVQOQK2SCENPB6RMYF2HUBKWVOGQA6N5RUFBDI2UA, seg_pos={0 0}, upload=133302]: Done uploading piece. err: ecclient: upload failed (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR, address:77.90.31.19:4870): protocol: rpc: dial tcp 77.90.31.19:4870: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:40:28.216942388 (128h12m29.04465079s) [noise=true , address=77.90.31.12:4603 , piece_id=QZHKCKRHMKFECKN3JK6XBSMHYT75PYDTYHP5LRPVNSSI5TTIWUDQ, seg_pos={0 0}, upload=133302]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: rpc: dial tcp 77.90.31.12:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:40:28.217144131 (128h12m29.044852531s) [noise=true , address=77.90.31.16:4738 , piece_id=2K2L3SRXBCLEFLJ45CZVN7UMYWIE7V7RJSAYSCP5KHJHAU4IN42Q, seg_pos={0 0}, upload=133302]: Done uploading piece. err: ecclient: upload failed (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd, address:77.90.31.16:4738): protocol: rpc: dial tcp 77.90.31.16:4738: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:40:28.217354380 (128h12m29.04506278s) [noise=true , address=77.90.31.17:4787 , piece_id=2APBQIBNWIWGHZYAIYNDGMDTNXGJWRU6JELGAQCSTW3FDHHXQLJQ, seg_pos={0 0}, upload=133302]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: rpc: dial tcp 77.90.31.17:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:40:28.217499055 (128h12m29.045207445s) [noise=true , address=77.90.31.13:4658 , piece_id=T6VI4T5UGDK67KY6URNU2G4RZPRAPQ2NRHPYBZNCXPFF7O5FIF3Q, seg_pos={0 0}, upload=133302]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: write tcp 159.223.143.39:38054->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:40:28.217676129 (128h12m29.045384528s) [noise=true , address=77.90.31.21:4896 , piece_id=HSU2N276S5IBLYZ2XBLS6X5LR4ETV5HVS6BPKEOKEE4FHB3H4BQA, seg_pos={0 0}, upload=133302]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; context deadline exceeded
Dec 10 23:40:28.217720361 (128h12m29.045428752s) [seg_pos={0 0}, upload=133302]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:40:28.240830538 (128h12m29.068538915s) [upload=133302]: Done uploading segments...
Dec 10 23:40:28.240863166 (128h12m29.068571558s) [upload=133302]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1m0.026179173s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [....................................X....X...X........X...X.] duration:
463.272988ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:40:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 463.295026ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....................................X....X...X........X...X] duration:
30.000570672s trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:40:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000593769s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.....................................XXX.X..................] duration:
552.23722ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:40:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 552.263571ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
14 out of 60 is failed (threshold=3) [..........................................XX..XXXXXX.XXXXXX.] duration:
458.871383ms trace:
https://jaeger.dev.storj.io/trace/1dfa88ce093f9a1d
FAIL (2025-12-10 23:39:59 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Dec 10 23:38:46.955133118 (128h10m47.782841516s) [upload=133290]: Starting upload
Dec 10 23:38:46.955199791 (128h10m47.782908182s) [upload=133290]: Done starting upload
Dec 10 23:38:46.955410807 (128h10m47.783119188s) [upload=133290]: Uploading segments...
Dec 10 23:38:46.955533284 (128h10m47.783241664s) [upload=133290]: Got next segment. Inline: false
Dec 10 23:38:46.955595804 (128h10m47.783304187s) [upload=133290]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:38:46.955602077 (128h10m47.783310468s) [upload=133290]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:38:46.983809626 (128h10m47.811518007s) [seg_pos={0 0}, upload=133290]: Begin upload segment...
Dec 10 23:38:46.983870989 (128h10m47.811579372s) [seg_pos={0 0}, upload=133290]: Done begin upload segment.
Dec 10 23:38:46.983927562 (128h10m47.811635956s) [noise=true , address=77.90.31.11:4578 , piece_id=SHSY5EBOAN7D2D4KL7PUUKYFKGXYZZB2IVZ64I7BN75X4RCQGZUA, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.983941540 (128h10m47.811649942s) [noise=true , address=45.91.116.11:4146 , piece_id=UC2BQIXQVTE7EYAA6SFQ4EMRQSU5IDY2M6IZST7E7NMOL2N3LKNA, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.984729337 (128h10m47.812437737s) [noise=true , address=145.79.227.31:4820 , piece_id=CP4RIALVPXO2O2XDMDIN6OBBCTBJ7X73246CTHWX2WA6RNMWQTFQ, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.984825248 (128h10m47.812533655s) [noise=true , address=145.79.227.52:4451 , piece_id=O5XAMDDBH46NUSADADTBL3GHBCKI5TA6CISMMVUEGCTHXQLNLO5A, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.985373395 (128h10m47.813081777s) [noise=true , address=77.90.31.17:4783 , piece_id=WIKNDVTNU5FRQTQED5DYWVSKTHZCDMMDLETA2NP3S2AUA35C3VGQ, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.985830177 (128h10m47.813538571s) [noise=true , address=5.231.73.5:4005 , piece_id=PJX5RUAOGL4FJAOJ6P2D3AGCH4PRQ4FJD5W4DRVLWS6TFPQXBGGA, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.986229940 (128h10m47.813938325s) [noise=true , address=45.91.116.6:4994 , piece_id=33TUOTF5IAHXDMWROKAAT6PMJXERQ3XP3VXCOPUYLC4BJ3LPBT4A, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.986660221 (128h10m47.814368615s) [noise=true , address=45.91.116.4:4966 , piece_id=XIB5KS3PJASPWMMSVOLDTZ35TQLR6TF36ES7HUMTMAZ3J7LFTJVQ, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.987158777 (128h10m47.814867171s) [noise=true , address=77.90.31.14:4665 , piece_id=A5S47WZVCNNYAMQIQWCIWVHBG5IAZ74FWP2KWJK54BC2PYAU576A, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.987547512 (128h10m47.81525591s) [noise=true , address=145.79.227.30:4770 , piece_id=YQBTZKHQI3AKKKVUX3XYLMBZJLXNBF6BVQKIWVUIOXQ6NNJYMH4A, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.987690015 (128h10m47.815398397s) [noise=true , address=145.79.227.32:4859 , piece_id=TTYKFTX5S3OPQ45Q75CMIR2FSIV6E3WQQ6UBL2DAPC2QHVNBH4NQ, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.988018046 (128h10m47.815726428s) [noise=true , address=45.91.116.10:4107 , piece_id=HJ7PIZUNJQLQIM2MLDDVXOABHECASBAV6LZRVS5HRM7T64UX3ZLQ, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.988256649 (128h10m47.815965046s) [noise=true , address=45.91.116.13:4182 , piece_id=NT4Z4IKIXAHEDT5ZLTQIB6GSSNGUHX6WMHXHYDDNUOT3EN57HUHQ, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.989467148 (128h10m47.817175542s) [noise=true , address=77.90.31.18:4824 , piece_id=IAYXCV6LBF3HNNM7NX42MA6GM7GVXMGTK5WDNICFSYULXYETRLYA, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.991205747 (128h10m47.818914147s) [noise=true , address=77.90.31.10:4550 , piece_id=WEGFFHAWQV45WKSOKPVVJJRVE456XNWKSEVO3A4NCJIBSIRUWMNA, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.992400274 (128h10m47.820108674s) [noise=true , address=77.90.31.21:4880 , piece_id=GHBKB2XDZHJWFLFILHBNVBMPDZZNAVZBOILXA75DZUJEGMV6CJ4Q, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.993810758 (128h10m47.821519158s) [noise=true , address=145.79.227.39:4043 , piece_id=ES34SHEJHTHX7V2EQEDF4UMINLEOXDOIXVIDPCSCHYG2COUPKWYQ, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.995387100 (128h10m47.823095501s) [noise=true , address=145.79.227.20:4472 , piece_id=ADJLALO2ZGJPW5SMXLZ6GKIWDGRXCMOV7B5PIKQWIUQMBYYXZNVQ, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:47.064146551 (128h10m47.891854943s) [upload=133290]: Next returned nil segment
Dec 10 23:38:47.064162076 (128h10m47.891870459s) [upload=133290]: Waiting for error group managing segments...
Dec 10 23:38:47.067552822 (128h10m47.895261213s) [noise=true , address=45.91.116.6:4994 , piece_id=33TUOTF5IAHXDMWROKAAT6PMJXERQ3XP3VXCOPUYLC4BJ3LPBT4A, seg_pos={0 0}, upload=133290]: Done uploading piece. err: <nil>
Dec 10 23:38:47.067638549 (128h10m47.895346931s) [noise=true , address=45.91.116.10:4107 , piece_id=HJ7PIZUNJQLQIM2MLDDVXOABHECASBAV6LZRVS5HRM7T64UX3ZLQ, seg_pos={0 0}, upload=133290]: Done uploading piece. err: <nil>
Dec 10 23:38:47.069005986 (128h10m47.896714379s) [noise=true , address=145.79.227.32:4859 , piece_id=TTYKFTX5S3OPQ45Q75CMIR2FSIV6E3WQQ6UBL2DAPC2QHVNBH4NQ, seg_pos={0 0}, upload=133290]: Done uploading piece. err: <nil>
Dec 10 23:38:47.069615509 (128h10m47.897323891s) [noise=true , address=145.79.227.39:4043 , piece_id=ES34SHEJHTHX7V2EQEDF4UMINLEOXDOIXVIDPCSCHYG2COUPKWYQ, seg_pos={0 0}, upload=133290]: Done uploading piece. err: <nil>
Dec 10 23:38:47.072736098 (128h10m47.900444482s) [noise=true , address=145.79.227.20:4472 , piece_id=ADJLALO2ZGJPW5SMXLZ6GKIWDGRXCMOV7B5PIKQWIUQMBYYXZNVQ, seg_pos={0 0}, upload=133290]: Done uploading piece. err: <nil>
Dec 10 23:38:47.077547046 (128h10m47.905255439s) [noise=true , address=45.91.116.13:4182 , piece_id=NT4Z4IKIXAHEDT5ZLTQIB6GSSNGUHX6WMHXHYDDNUOT3EN57HUHQ, seg_pos={0 0}, upload=133290]: Done uploading piece. err: <nil>
Dec 10 23:38:47.083428630 (128h10m47.911137021s) [noise=true , address=45.91.116.4:4966 , piece_id=XIB5KS3PJASPWMMSVOLDTZ35TQLR6TF36ES7HUMTMAZ3J7LFTJVQ, seg_pos={0 0}, upload=133290]: Done uploading piece. err: <nil>
Dec 10 23:38:47.085956871 (128h10m47.913665268s) [noise=true , address=45.91.116.11:4146 , piece_id=UC2BQIXQVTE7EYAA6SFQ4EMRQSU5IDY2M6IZST7E7NMOL2N3LKNA, seg_pos={0 0}, upload=133290]: Done uploading piece. err: <nil>
Dec 10 23:38:47.097921479 (128h10m47.925629874s) [noise=true , address=145.79.227.31:4820 , piece_id=CP4RIALVPXO2O2XDMDIN6OBBCTBJ7X73246CTHWX2WA6RNMWQTFQ, seg_pos={0 0}, upload=133290]: Done uploading piece. err: <nil>
Dec 10 23:38:47.103831932 (128h10m47.931540323s) [noise=true , address=145.79.227.52:4451 , piece_id=O5XAMDDBH46NUSADADTBL3GHBCKI5TA6CISMMVUEGCTHXQLNLO5A, seg_pos={0 0}, upload=133290]: Done uploading piece. err: <nil>
Dec 10 23:38:47.165887586 (128h10m47.993595977s) [noise=true , address=5.231.73.5:4005 , piece_id=PJX5RUAOGL4FJAOJ6P2D3AGCH4PRQ4FJD5W4DRVLWS6TFPQXBGGA, seg_pos={0 0}, upload=133290]: Done uploading piece. err: <nil>
Dec 10 23:38:47.999173193 (128h10m48.826881575s) [noise=true , address=145.79.227.30:4770 , piece_id=YQBTZKHQI3AKKKVUX3XYLMBZJLXNBF6BVQKIWVUIOXQ6NNJYMH4A, seg_pos={0 0}, upload=133290]: Done uploading piece. err: <nil>
Dec 10 23:39:46.918420745 (128h11m47.746129142s) [noise=true , address=77.90.31.10:4550 , piece_id=WEGFFHAWQV45WKSOKPVVJJRVE456XNWKSEVO3A4NCJIBSIRUWMNA, seg_pos={0 0}, upload=133290]: Done uploading piece. err: ecclient: upload failed (node:12nfBW1W3RvEGzRz2bYA1hGxMfvsxVR6WNn5oPUdtJU26KMXNE7, address:77.90.31.10:4550): protocol: write tcp 159.223.143.39:35508->77.90.31.10:4550: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:39:46.918584261 (128h11m47.746292661s) [noise=true , address=77.90.31.17:4783 , piece_id=WIKNDVTNU5FRQTQED5DYWVSKTHZCDMMDLETA2NP3S2AUA35C3VGQ, seg_pos={0 0}, upload=133290]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:39:46.918621279 (128h11m47.746329681s) [noise=true , address=77.90.31.21:4880 , piece_id=GHBKB2XDZHJWFLFILHBNVBMPDZZNAVZBOILXA75DZUJEGMV6CJ4Q, seg_pos={0 0}, upload=133290]: Done uploading piece. err: ecclient: upload failed (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, address:77.90.31.21:4880): protocol: rpc: dial tcp 77.90.31.21:4880: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:39:46.918963283 (128h11m47.746671692s) [noise=true , address=77.90.31.18:4824 , piece_id=IAYXCV6LBF3HNNM7NX42MA6GM7GVXMGTK5WDNICFSYULXYETRLYA, seg_pos={0 0}, upload=133290]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:39:46.919297833 (128h11m47.747006237s) [noise=true , address=77.90.31.14:4665 , piece_id=A5S47WZVCNNYAMQIQWCIWVHBG5IAZ74FWP2KWJK54BC2PYAU576A, seg_pos={0 0}, upload=133290]: Done uploading piece. err: ecclient: upload failed (node:12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, address:77.90.31.14:4665): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:39:46.919339769 (128h11m47.747048163s) [noise=true , address=77.90.31.11:4578 , piece_id=SHSY5EBOAN7D2D4KL7PUUKYFKGXYZZB2IVZ64I7BN75X4RCQGZUA, seg_pos={0 0}, upload=133290]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: write tcp 159.223.143.39:48602->77.90.31.11:4578: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:39:46.919366281 (128h11m47.747074666s) [seg_pos={0 0}, upload=133290]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:39:46.947126305 (128h11m47.774834696s) [upload=133290]: Done uploading segments...
Dec 10 23:39:46.947162600 (128h11m47.774870991s) [upload=133290]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 458.902792ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [......................................XXX...................] duration:
572.66417ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:39:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 572.694477ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
14 out of 60 is failed (threshold=3) [...........................................XX..XXXXXX.XXXXXX] duration:
1m0.030595013s trace:
https://jaeger.dev.storj.io/trace/1dfa88ce093f9a1d
FAIL (2025-12-10 23:39:46 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Dec 10 23:38:46.955133118 (128h10m47.782841516s) [upload=133290]: Starting upload
Dec 10 23:38:46.955199791 (128h10m47.782908182s) [upload=133290]: Done starting upload
Dec 10 23:38:46.955410807 (128h10m47.783119188s) [upload=133290]: Uploading segments...
Dec 10 23:38:46.955533284 (128h10m47.783241664s) [upload=133290]: Got next segment. Inline: false
Dec 10 23:38:46.955595804 (128h10m47.783304187s) [upload=133290]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:38:46.955602077 (128h10m47.783310468s) [upload=133290]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:38:46.983809626 (128h10m47.811518007s) [seg_pos={0 0}, upload=133290]: Begin upload segment...
Dec 10 23:38:46.983870989 (128h10m47.811579372s) [seg_pos={0 0}, upload=133290]: Done begin upload segment.
Dec 10 23:38:46.983927562 (128h10m47.811635956s) [noise=true , address=77.90.31.11:4578 , piece_id=SHSY5EBOAN7D2D4KL7PUUKYFKGXYZZB2IVZ64I7BN75X4RCQGZUA, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.983941540 (128h10m47.811649942s) [noise=true , address=45.91.116.11:4146 , piece_id=UC2BQIXQVTE7EYAA6SFQ4EMRQSU5IDY2M6IZST7E7NMOL2N3LKNA, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.984729337 (128h10m47.812437737s) [noise=true , address=145.79.227.31:4820 , piece_id=CP4RIALVPXO2O2XDMDIN6OBBCTBJ7X73246CTHWX2WA6RNMWQTFQ, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.984825248 (128h10m47.812533655s) [noise=true , address=145.79.227.52:4451 , piece_id=O5XAMDDBH46NUSADADTBL3GHBCKI5TA6CISMMVUEGCTHXQLNLO5A, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.985373395 (128h10m47.813081777s) [noise=true , address=77.90.31.17:4783 , piece_id=WIKNDVTNU5FRQTQED5DYWVSKTHZCDMMDLETA2NP3S2AUA35C3VGQ, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.985830177 (128h10m47.813538571s) [noise=true , address=5.231.73.5:4005 , piece_id=PJX5RUAOGL4FJAOJ6P2D3AGCH4PRQ4FJD5W4DRVLWS6TFPQXBGGA, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.986229940 (128h10m47.813938325s) [noise=true , address=45.91.116.6:4994 , piece_id=33TUOTF5IAHXDMWROKAAT6PMJXERQ3XP3VXCOPUYLC4BJ3LPBT4A, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.986660221 (128h10m47.814368615s) [noise=true , address=45.91.116.4:4966 , piece_id=XIB5KS3PJASPWMMSVOLDTZ35TQLR6TF36ES7HUMTMAZ3J7LFTJVQ, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.987158777 (128h10m47.814867171s) [noise=true , address=77.90.31.14:4665 , piece_id=A5S47WZVCNNYAMQIQWCIWVHBG5IAZ74FWP2KWJK54BC2PYAU576A, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.987547512 (128h10m47.81525591s) [noise=true , address=145.79.227.30:4770 , piece_id=YQBTZKHQI3AKKKVUX3XYLMBZJLXNBF6BVQKIWVUIOXQ6NNJYMH4A, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.987690015 (128h10m47.815398397s) [noise=true , address=145.79.227.32:4859 , piece_id=TTYKFTX5S3OPQ45Q75CMIR2FSIV6E3WQQ6UBL2DAPC2QHVNBH4NQ, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.988018046 (128h10m47.815726428s) [noise=true , address=45.91.116.10:4107 , piece_id=HJ7PIZUNJQLQIM2MLDDVXOABHECASBAV6LZRVS5HRM7T64UX3ZLQ, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.988256649 (128h10m47.815965046s) [noise=true , address=45.91.116.13:4182 , piece_id=NT4Z4IKIXAHEDT5ZLTQIB6GSSNGUHX6WMHXHYDDNUOT3EN57HUHQ, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.989467148 (128h10m47.817175542s) [noise=true , address=77.90.31.18:4824 , piece_id=IAYXCV6LBF3HNNM7NX42MA6GM7GVXMGTK5WDNICFSYULXYETRLYA, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.991205747 (128h10m47.818914147s) [noise=true , address=77.90.31.10:4550 , piece_id=WEGFFHAWQV45WKSOKPVVJJRVE456XNWKSEVO3A4NCJIBSIRUWMNA, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.992400274 (128h10m47.820108674s) [noise=true , address=77.90.31.21:4880 , piece_id=GHBKB2XDZHJWFLFILHBNVBMPDZZNAVZBOILXA75DZUJEGMV6CJ4Q, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.993810758 (128h10m47.821519158s) [noise=true , address=145.79.227.39:4043 , piece_id=ES34SHEJHTHX7V2EQEDF4UMINLEOXDOIXVIDPCSCHYG2COUPKWYQ, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:46.995387100 (128h10m47.823095501s) [noise=true , address=145.79.227.20:4472 , piece_id=ADJLALO2ZGJPW5SMXLZ6GKIWDGRXCMOV7B5PIKQWIUQMBYYXZNVQ, seg_pos={0 0}, upload=133290]: Uploading piece...
Dec 10 23:38:47.064146551 (128h10m47.891854943s) [upload=133290]: Next returned nil segment
Dec 10 23:38:47.064162076 (128h10m47.891870459s) [upload=133290]: Waiting for error group managing segments...
Dec 10 23:38:47.067552822 (128h10m47.895261213s) [noise=true , address=45.91.116.6:4994 , piece_id=33TUOTF5IAHXDMWROKAAT6PMJXERQ3XP3VXCOPUYLC4BJ3LPBT4A, seg_pos={0 0}, upload=133290]: Done uploading piece. err: <nil>
Dec 10 23:38:47.067638549 (128h10m47.895346931s) [noise=true , address=45.91.116.10:4107 , piece_id=HJ7PIZUNJQLQIM2MLDDVXOABHECASBAV6LZRVS5HRM7T64UX3ZLQ, seg_pos={0 0}, upload=133290]: Done uploading piece. err: <nil>
Dec 10 23:38:47.069005986 (128h10m47.896714379s) [noise=true , address=145.79.227.32:4859 , piece_id=TTYKFTX5S3OPQ45Q75CMIR2FSIV6E3WQQ6UBL2DAPC2QHVNBH4NQ, seg_pos={0 0}, upload=133290]: Done uploading piece. err: <nil>
Dec 10 23:38:47.069615509 (128h10m47.897323891s) [noise=true , address=145.79.227.39:4043 , piece_id=ES34SHEJHTHX7V2EQEDF4UMINLEOXDOIXVIDPCSCHYG2COUPKWYQ, seg_pos={0 0}, upload=133290]: Done uploading piece. err: <nil>
Dec 10 23:38:47.072736098 (128h10m47.900444482s) [noise=true , address=145.79.227.20:4472 , piece_id=ADJLALO2ZGJPW5SMXLZ6GKIWDGRXCMOV7B5PIKQWIUQMBYYXZNVQ, seg_pos={0 0}, upload=133290]: Done uploading piece. err: <nil>
Dec 10 23:38:47.077547046 (128h10m47.905255439s) [noise=true , address=45.91.116.13:4182 , piece_id=NT4Z4IKIXAHEDT5ZLTQIB6GSSNGUHX6WMHXHYDDNUOT3EN57HUHQ, seg_pos={0 0}, upload=133290]: Done uploading piece. err: <nil>
Dec 10 23:38:47.083428630 (128h10m47.911137021s) [noise=true , address=45.91.116.4:4966 , piece_id=XIB5KS3PJASPWMMSVOLDTZ35TQLR6TF36ES7HUMTMAZ3J7LFTJVQ, seg_pos={0 0}, upload=133290]: Done uploading piece. err: <nil>
Dec 10 23:38:47.085956871 (128h10m47.913665268s) [noise=true , address=45.91.116.11:4146 , piece_id=UC2BQIXQVTE7EYAA6SFQ4EMRQSU5IDY2M6IZST7E7NMOL2N3LKNA, seg_pos={0 0}, upload=133290]: Done uploading piece. err: <nil>
Dec 10 23:38:47.097921479 (128h10m47.925629874s) [noise=true , address=145.79.227.31:4820 , piece_id=CP4RIALVPXO2O2XDMDIN6OBBCTBJ7X73246CTHWX2WA6RNMWQTFQ, seg_pos={0 0}, upload=133290]: Done uploading piece. err: <nil>
Dec 10 23:38:47.103831932 (128h10m47.931540323s) [noise=true , address=145.79.227.52:4451 , piece_id=O5XAMDDBH46NUSADADTBL3GHBCKI5TA6CISMMVUEGCTHXQLNLO5A, seg_pos={0 0}, upload=133290]: Done uploading piece. err: <nil>
Dec 10 23:38:47.165887586 (128h10m47.993595977s) [noise=true , address=5.231.73.5:4005 , piece_id=PJX5RUAOGL4FJAOJ6P2D3AGCH4PRQ4FJD5W4DRVLWS6TFPQXBGGA, seg_pos={0 0}, upload=133290]: Done uploading piece. err: <nil>
Dec 10 23:38:47.999173193 (128h10m48.826881575s) [noise=true , address=145.79.227.30:4770 , piece_id=YQBTZKHQI3AKKKVUX3XYLMBZJLXNBF6BVQKIWVUIOXQ6NNJYMH4A, seg_pos={0 0}, upload=133290]: Done uploading piece. err: <nil>
Dec 10 23:39:46.918420745 (128h11m47.746129142s) [noise=true , address=77.90.31.10:4550 , piece_id=WEGFFHAWQV45WKSOKPVVJJRVE456XNWKSEVO3A4NCJIBSIRUWMNA, seg_pos={0 0}, upload=133290]: Done uploading piece. err: ecclient: upload failed (node:12nfBW1W3RvEGzRz2bYA1hGxMfvsxVR6WNn5oPUdtJU26KMXNE7, address:77.90.31.10:4550): protocol: write tcp 159.223.143.39:35508->77.90.31.10:4550: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:39:46.918584261 (128h11m47.746292661s) [noise=true , address=77.90.31.17:4783 , piece_id=WIKNDVTNU5FRQTQED5DYWVSKTHZCDMMDLETA2NP3S2AUA35C3VGQ, seg_pos={0 0}, upload=133290]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:39:46.918621279 (128h11m47.746329681s) [noise=true , address=77.90.31.21:4880 , piece_id=GHBKB2XDZHJWFLFILHBNVBMPDZZNAVZBOILXA75DZUJEGMV6CJ4Q, seg_pos={0 0}, upload=133290]: Done uploading piece. err: ecclient: upload failed (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, address:77.90.31.21:4880): protocol: rpc: dial tcp 77.90.31.21:4880: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:39:46.918963283 (128h11m47.746671692s) [noise=true , address=77.90.31.18:4824 , piece_id=IAYXCV6LBF3HNNM7NX42MA6GM7GVXMGTK5WDNICFSYULXYETRLYA, seg_pos={0 0}, upload=133290]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:39:46.919297833 (128h11m47.747006237s) [noise=true , address=77.90.31.14:4665 , piece_id=A5S47WZVCNNYAMQIQWCIWVHBG5IAZ74FWP2KWJK54BC2PYAU576A, seg_pos={0 0}, upload=133290]: Done uploading piece. err: ecclient: upload failed (node:12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, address:77.90.31.14:4665): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:39:46.919339769 (128h11m47.747048163s) [noise=true , address=77.90.31.11:4578 , piece_id=SHSY5EBOAN7D2D4KL7PUUKYFKGXYZZB2IVZ64I7BN75X4RCQGZUA, seg_pos={0 0}, upload=133290]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: write tcp 159.223.143.39:48602->77.90.31.11:4578: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:39:46.919366281 (128h11m47.747074666s) [seg_pos={0 0}, upload=133290]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:39:46.947126305 (128h11m47.774834696s) [upload=133290]: Done uploading segments...
Dec 10 23:39:46.947162600 (128h11m47.774870991s) [upload=133290]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1m0.030627434s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [......................................XXX.X.................] duration:
488.492602ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:39:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 488.533875ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
15 out of 60 is failed (threshold=3) [............................................XXXXXXX.XXXXXXXX] duration:
1m0.026052517s trace:
https://jaeger.dev.storj.io/trace/4c97daaadc5c6ad5
FAIL (2025-12-10 23:39:28 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
7 (128h10m28.987616799s) [upload=133285]: Got next segment. Inline: false
Dec 10 23:38:28.159964770 (128h10m28.987673151s) [upload=133285]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:38:28.159970025 (128h10m28.987678407s) [upload=133285]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:38:28.190369149 (128h10m29.018077543s) [seg_pos={0 0}, upload=133285]: Begin upload segment...
Dec 10 23:38:28.190426030 (128h10m29.018134412s) [seg_pos={0 0}, upload=133285]: Done begin upload segment.
Dec 10 23:38:28.190452117 (128h10m29.018160519s) [noise=true , address=145.79.227.48:4330 , piece_id=ELETPAXHCGVIZGZ363CJGLZOEET35NXTOKHSBLMAYQGIT2NBD2WQ, seg_pos={0 0}, upload=133285]: Uploading piece...
Dec 10 23:38:28.190467763 (128h10m29.018176154s) [noise=true , address=77.90.31.16:4758 , piece_id=SRLFSCJMIJVT4KTO4WJSFLKMGHOLYS3KXXXRA6B3WDBJ2V2QNZ7Q, seg_pos={0 0}, upload=133285]: Uploading piece...
Dec 10 23:38:28.191243928 (128h10m29.018952328s) [noise=true , address=145.79.227.6:4290 , piece_id=RRDWFBM676NLXCZ4IAL225RBFNCEH5MUTFKHKSSAN4BKYU5ZRISA, seg_pos={0 0}, upload=133285]: Uploading piece...
Dec 10 23:38:28.191734972 (128h10m29.019443357s) [noise=true , address=145.79.227.15:4903 , piece_id=GRMTIZX5FRBVD6YNPSKU7TLX4CEFEWN3UVGIO2ZT6QH3W65FUMXQ, seg_pos={0 0}, upload=133285]: Uploading piece...
Dec 10 23:38:28.191809275 (128h10m29.019517675s) [noise=true , address=77.90.31.21:4890 , piece_id=D6BBQQLNZN2C23F6TGJQPEWSIZ7AXPIQ5VVBNCNVCIQSUWHQJWNQ, seg_pos={0 0}, upload=133285]: Uploading piece...
Dec 10 23:38:28.192265764 (128h10m29.01997416s) [noise=true , address=145.79.227.44:4199 , piece_id=3SSUYVERRDAQGGVG2Y4SKATPYBBIDGJZDPM3LNCEYJAFI4SF2LBA, seg_pos={0 0}, upload=133285]: Uploading piece...
Dec 10 23:38:28.192792925 (128h10m29.020501319s) [noise=true , address=77.90.31.11:4586 , piece_id=XK2JJLWEKYAWOSPLLDFRIJZKS7D5YXWARPM4TNDO5G7NSGOXLT7A, seg_pos={0 0}, upload=133285]: Uploading piece...
Dec 10 23:38:28.193182040 (128h10m29.020890432s) [noise=true , address=145.79.227.16:4939 , piece_id=JGJPF2PJHQVXYNLJGKLTAOV4VRTE3E73QDZ2IUM4KMOLS6JRLM6Q, seg_pos={0 0}, upload=133285]: Uploading piece...
Dec 10 23:38:28.193591000 (128h10m29.021299393s) [noise=true , address=5.231.73.3:4032 , piece_id=E4ZZ2332WB7VEPPNGUVEG7C4RDXP3OQ5DXVURZE2VIJR4DUAIU4A, seg_pos={0 0}, upload=133285]: Uploading piece...
Dec 10 23:38:28.194037172 (128h10m29.021745563s) [noise=true , address=45.91.116.12:4238 , piece_id=N5Z6CXAEKHO2GOTJJRDSSJBCGBJOY3MGUHHWTQH62UNQ6TIVQ7NA, seg_pos={0 0}, upload=133285]: Uploading piece...
Dec 10 23:38:28.194473906 (128h10m29.022182301s) [noise=true , address=5.231.73.5:4004 , piece_id=OAZBBQINDSKO4A5MR5KC55RWCXWU5QPTDS7KMZ3J7ZEKQEHZYCBQ, seg_pos={0 0}, upload=133285]: Uploading piece...
Dec 10 23:38:28.195013922 (128h10m29.022722319s) [noise=true , address=77.90.31.17:4799 , piece_id=QZIFFOIF3E57Z3PUSKNJNYIMGTME7GKZ4HXZFYG3S6GLURDQSSSA, seg_pos={0 0}, upload=133285]: Uploading piece...
Dec 10 23:38:28.195563790 (128h10m29.023272182s) [noise=true , address=45.91.116.8:4073 , piece_id=BY7QHNXIQSN5ZDEWOCWDT3Y5K4UUFHNYTYPKQNAJ6RJW424QJFZQ, seg_pos={0 0}, upload=133285]: Uploading piece...
Dec 10 23:38:28.196172113 (128h10m29.023880504s) [noise=true , address=145.79.227.33:4891 , piece_id=VCIY4NIONQACJLHDSZHLHJQFBDFLMYV3P3BIX4TIEN4Y2QDZXMBQ, seg_pos={0 0}, upload=133285]: Uploading piece...
Dec 10 23:38:28.196689230 (128h10m29.02439763s) [noise=true , address=145.79.227.30:4784 , piece_id=QZINURJO45JKDIAW3H77JTNEM2CX6FAQNH2JQZ766QD5MO26P4AQ, seg_pos={0 0}, upload=133285]: Uploading piece...
Dec 10 23:38:28.197281612 (128h10m29.024990006s) [noise=true , address=77.90.31.14:4668 , piece_id=5H62F75AWMSPFQXM5SNUOVFPPIQMGF56V25CPHDZYNJ4TJKOFR6Q, seg_pos={0 0}, upload=133285]: Uploading piece...
Dec 10 23:38:28.197812484 (128h10m29.025520878s) [noise=true , address=45.91.116.12:4247 , piece_id=NUA4DMNRVX7LIETX6GCNXRWLOAGT7QHMAHLROOEHGF6BJIEFYN3A, seg_pos={0 0}, upload=133285]: Uploading piece...
Dec 10 23:38:28.198355537 (128h10m29.026063934s) [noise=true , address=64.49.13.27:4964 , piece_id=TXHKSFT6M2GGI4DWQU34PAQFMHKQ6OVHYP4R6PLRJ2PGYIWQKAZA, seg_pos={0 0}, upload=133285]: Uploading piece...
Dec 10 23:38:28.268270587 (128h10m29.09597897s) [upload=133285]: Next returned nil segment
Dec 10 23:38:28.268284155 (128h10m29.095992536s) [upload=133285]: Waiting for error group managing segments...
Dec 10 23:38:28.271689774 (128h10m29.099398165s) [noise=true , address=45.91.116.12:4247 , piece_id=NUA4DMNRVX7LIETX6GCNXRWLOAGT7QHMAHLROOEHGF6BJIEFYN3A, seg_pos={0 0}, upload=133285]: Done uploading piece. err: <nil>
Dec 10 23:38:28.271999649 (128h10m29.099708047s) [noise=true , address=45.91.116.12:4238 , piece_id=N5Z6CXAEKHO2GOTJJRDSSJBCGBJOY3MGUHHWTQH62UNQ6TIVQ7NA, seg_pos={0 0}, upload=133285]: Done uploading piece. err: <nil>
Dec 10 23:38:28.275293607 (128h10m29.103001999s) [noise=true , address=145.79.227.33:4891 , piece_id=VCIY4NIONQACJLHDSZHLHJQFBDFLMYV3P3BIX4TIEN4Y2QDZXMBQ, seg_pos={0 0}, upload=133285]: Done uploading piece. err: <nil>
Dec 10 23:38:28.286700295 (128h10m29.114408687s) [noise=true , address=145.79.227.6:4290 , piece_id=RRDWFBM676NLXCZ4IAL225RBFNCEH5MUTFKHKSSAN4BKYU5ZRISA, seg_pos={0 0}, upload=133285]: Done uploading piece. err: <nil>
Dec 10 23:38:28.294860632 (128h10m29.122569024s) [noise=true , address=45.91.116.8:4073 , piece_id=BY7QHNXIQSN5ZDEWOCWDT3Y5K4UUFHNYTYPKQNAJ6RJW424QJFZQ, seg_pos={0 0}, upload=133285]: Done uploading piece. err: <nil>
Dec 10 23:38:28.295819722 (128h10m29.123528113s) [noise=true , address=145.79.227.48:4330 , piece_id=ELETPAXHCGVIZGZ363CJGLZOEET35NXTOKHSBLMAYQGIT2NBD2WQ, seg_pos={0 0}, upload=133285]: Done uploading piece. err: <nil>
Dec 10 23:38:28.296027048 (128h10m29.123735442s) [noise=true , address=145.79.227.15:4903 , piece_id=GRMTIZX5FRBVD6YNPSKU7TLX4CEFEWN3UVGIO2ZT6QH3W65FUMXQ, seg_pos={0 0}, upload=133285]: Done uploading piece. err: <nil>
Dec 10 23:38:28.357003968 (128h10m29.184712359s) [noise=true , address=5.231.73.3:4032 , piece_id=E4ZZ2332WB7VEPPNGUVEG7C4RDXP3OQ5DXVURZE2VIJR4DUAIU4A, seg_pos={0 0}, upload=133285]: Done uploading piece. err: <nil>
Dec 10 23:38:28.566794538 (128h10m29.394502929s) [noise=true , address=5.231.73.5:4004 , piece_id=OAZBBQINDSKO4A5MR5KC55RWCXWU5QPTDS7KMZ3J7ZEKQEHZYCBQ, seg_pos={0 0}, upload=133285]: Done uploading piece. err: <nil>
Dec 10 23:38:28.791250895 (128h10m29.618959286s) [noise=true , address=64.49.13.27:4964 , piece_id=TXHKSFT6M2GGI4DWQU34PAQFMHKQ6OVHYP4R6PLRJ2PGYIWQKAZA, seg_pos={0 0}, upload=133285]: Done uploading piece. err: <nil>
Dec 10 23:38:29.207993067 (128h10m30.035701465s) [noise=true , address=145.79.227.44:4199 , piece_id=3SSUYVERRDAQGGVG2Y4SKATPYBBIDGJZDPM3LNCEYJAFI4SF2LBA, seg_pos={0 0}, upload=133285]: Done uploading piece. err: <nil>
Dec 10 23:38:29.257812073 (128h10m30.085520471s) [noise=true , address=145.79.227.30:4784 , piece_id=QZINURJO45JKDIAW3H77JTNEM2CX6FAQNH2JQZ766QD5MO26P4AQ, seg_pos={0 0}, upload=133285]: Done uploading piece. err: <nil>
Dec 10 23:38:29.311260299 (128h10m30.138968689s) [noise=true , address=145.79.227.16:4939 , piece_id=JGJPF2PJHQVXYNLJGKLTAOV4VRTE3E73QDZ2IUM4KMOLS6JRLM6Q, seg_pos={0 0}, upload=133285]: Done uploading piece. err: <nil>
Dec 10 23:39:28.130289815 (128h11m28.957998207s) [noise=true , address=77.90.31.11:4586 , piece_id=XK2JJLWEKYAWOSPLLDFRIJZKS7D5YXWARPM4TNDO5G7NSGOXLT7A, seg_pos={0 0}, upload=133285]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: rpc: dial tcp 77.90.31.11:4586: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:39:28.130340169 (128h11m28.958048566s) [noise=true , address=77.90.31.14:4668 , piece_id=5H62F75AWMSPFQXM5SNUOVFPPIQMGF56V25CPHDZYNJ4TJKOFR6Q, seg_pos={0 0}, upload=133285]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 159.223.143.39:42952->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:39:28.130550916 (128h11m28.958259314s) [noise=true , address=77.90.31.16:4758 , piece_id=SRLFSCJMIJVT4KTO4WJSFLKMGHOLYS3KXXXRA6B3WDBJ2V2QNZ7Q, seg_pos={0 0}, upload=133285]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: write tcp 159.223.143.39:50678->77.90.31.16:4758: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:39:28.130607058 (128h11m28.958315449s) [noise=true , address=77.90.31.17:4799 , piece_id=QZIFFOIF3E57Z3PUSKNJNYIMGTME7GKZ4HXZFYG3S6GLURDQSSSA, seg_pos={0 0}, upload=133285]: Done uploading piece. err: ecclient: upload failed (node:12UhAksgXZuFGAVheaYzP7D4kwVyAjUAq3BtrvdSxszjNCFCTWT, address:77.90.31.17:4799): protocol: rpc: dial tcp 77.90.31.17:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:39:28.130792668 (128h11m28.958501068s) [noise=true , address=77.90.31.21:4890 , piece_id=D6BBQQLNZN2C23F6TGJQPEWSIZ7AXPIQ5VVBNCNVCIQSUWHQJWNQ, seg_pos={0 0}, upload=133285]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: rpc: dial tcp 77.90.31.21:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:39:28.130823443 (128h11m28.958531825s) [seg_pos={0 0}, upload=133285]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 10 23:39:28.153260767 (128h11m28.980969161s) [upload=133285]: Done uploading segments...
Dec 10 23:39:28.153296256 (128h11m28.98100465s) [upload=133285]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1m0.026065925s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [......................................X....X...X........X...] duration:
506.722245ms trace:
https://jaeger.dev.storj.io/trace/2d7c696faa1ab964
FAIL (2025-12-10 23:39:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 506.737839ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.......................................XXX..................] duration:
530.27294ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:39:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 530.299631ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
13 out of 60 is failed (threshold=3) [............................................XX..XXXXXX.XXXXX] duration:
1m0.025071242s trace:
https://jaeger.dev.storj.io/trace/696210cfaebbcca
FAIL (2025-12-10 23:38:46 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Dec 10 23:37:46.858177117 (128h9m47.685885516s) [upload=133273]: Starting upload
Dec 10 23:37:46.858239344 (128h9m47.685947746s) [upload=133273]: Done starting upload
Dec 10 23:37:46.858661487 (128h9m47.68636988s) [upload=133273]: Uploading segments...
Dec 10 23:37:46.858725219 (128h9m47.68643362s) [upload=133273]: Got next segment. Inline: false
Dec 10 23:37:46.858767077 (128h9m47.686475476s) [upload=133273]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:37:46.858772978 (128h9m47.686481375s) [upload=133273]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:37:46.886027138 (128h9m47.71373553s) [seg_pos={0 0}, upload=133273]: Begin upload segment...
Dec 10 23:37:46.886097732 (128h9m47.713806116s) [seg_pos={0 0}, upload=133273]: Done begin upload segment.
Dec 10 23:37:46.886171496 (128h9m47.713879893s) [noise=true , address=45.91.116.9:4293 , piece_id=DXMEHZM2NGXM3ITINVK455ZJ2V2G6JNQUW3BBSKOVSNW3AH66HFA, seg_pos={0 0}, upload=133273]: Uploading piece...
Dec 10 23:37:46.886966307 (128h9m47.714674705s) [noise=true , address=77.90.31.12:4598 , piece_id=BG3NZFPB6L26IXA2NQALNWS3JHYYQDH763STNQ5OGWXLAA4L522A, seg_pos={0 0}, upload=133273]: Uploading piece...
Dec 10 23:37:46.887116387 (128h9m47.714824794s) [noise=true , address=5.231.73.3:4001 , piece_id=HCN4EP4VRVTIFKRDPIC7OBZMERZXDRUGJC2EM773WNCQX6VZRHNQ, seg_pos={0 0}, upload=133273]: Uploading piece...
Dec 10 23:37:46.887396681 (128h9m47.715105084s) [noise=true , address=145.79.227.3:4167 , piece_id=IOZDFIBKJYHMSUUI25Z4AMIHEKYMVR3UYGZRPCQ56FUHUZ662MHQ, seg_pos={0 0}, upload=133273]: Uploading piece...
Dec 10 23:37:46.887703480 (128h9m47.715411883s) [noise=true , address=45.91.116.13:4180 , piece_id=M2MLX3FEUCTSIFXNMWRYRF2HH4YGDWCKKZYGTIJTOJYXPOV7D7OA, seg_pos={0 0}, upload=133273]: Uploading piece...
Dec 10 23:37:46.887799552 (128h9m47.715507947s) [noise=true , address=45.91.116.7:4022 , piece_id=CL7ZVEOG3UYPJCSMYNAHTJCUTBZX5BV6SPLZKVEBGGE5UEEBXLEQ, seg_pos={0 0}, upload=133273]: Uploading piece...
Dec 10 23:37:46.888185772 (128h9m47.715894175s) [noise=true , address=45.91.116.14:4204 , piece_id=EKG3TWE3KQOJ23GPEPMRGQGGXJD4S3BO2G3TM3IYVCP7WIRQSNUA, seg_pos={0 0}, upload=133273]: Uploading piece...
Dec 10 23:37:46.888213226 (128h9m47.715921627s) [noise=true , address=45.91.116.5:4327 , piece_id=K3HHQ3P5NHK2VJ3MZ76O6JRY4NS4IMUMRR7UWIG7DCRBYY3GE2XA, seg_pos={0 0}, upload=133273]: Uploading piece...
Dec 10 23:37:46.888710104 (128h9m47.716418501s) [noise=true , address=45.91.116.11:4127 , piece_id=FF6Y6LNYGKTKLAQEIO5Q7XE5LBTVHIHMEORMQE5NHM6XLRCF73RA, seg_pos={0 0}, upload=133273]: Uploading piece...
Dec 10 23:37:46.888729443 (128h9m47.716437838s) [noise=true , address=77.90.31.21:4911 , piece_id=QW7TPOIM2FHHHHFHR6OO57IMRNPYUCIXPIRLC5BPNSFK4TCQZE3Q, seg_pos={0 0}, upload=133273]: Uploading piece...
Dec 10 23:37:46.889121546 (128h9m47.716829941s) [noise=true , address=77.90.31.13:4629 , piece_id=W7EGWRNIDJH37DGGXPR6674EKAA7MFXK7IL2TDLB63E3V6DVH6OA, seg_pos={0 0}, upload=133273]: Uploading piece...
Dec 10 23:37:46.889167036 (128h9m47.71687544s) [noise=true , address=145.79.227.48:4313 , piece_id=3YJGBWJU26CHIKQIMXGUDNL3HQESBMD5XOTPPG7FZG3UCU4JLZPQ, seg_pos={0 0}, upload=133273]: Uploading piece...
Dec 10 23:37:46.889485107 (128h9m47.717193499s) [noise=true , address=77.90.31.14:4673 , piece_id=ELTOW2L7HO27C3BRVJUJMSS2E5XKUUGYDPVN4CJ4O26MQV65HBHQ, seg_pos={0 0}, upload=133273]: Uploading piece...
Dec 10 23:37:46.889616163 (128h9m47.717324564s) [noise=true , address=5.231.73.4:4010 , piece_id=LZ3PZRDYY4HFFA27E5ELKBOITHDWD5XV4NHMQJPQ3FPAEC7VLJ3A, seg_pos={0 0}, upload=133273]: Uploading piece...
Dec 10 23:37:46.890163091 (128h9m47.717871496s) [noise=true , address=77.90.31.19:4869 , piece_id=SEWKXV6X2U24Q73ZW5WNAK7EIG2Z7VG3VWBVJA5KHHPVTY6UTA6Q, seg_pos={0 0}, upload=133273]: Uploading piece...
Dec 10 23:37:46.890653074 (128h9m47.718361477s) [noise=true , address=77.90.31.15:4705 , piece_id=DD7GJGQJ4CUFYRFGIUOACGKEYBCN6RML4XUO6DVHRVRA735JP37Q, seg_pos={0 0}, upload=133273]: Uploading piece...
Dec 10 23:37:46.891141942 (128h9m47.718850346s) [noise=true , address=45.91.116.6:4993 , piece_id=4WCSWAPCLQCGBDCF4CS2AFLYRANZZ3R6OPMNPMFRUMQB53SAI3GQ, seg_pos={0 0}, upload=133273]: Uploading piece...
Dec 10 23:37:46.892174078 (128h9m47.719882462s) [noise=true , address=77.90.31.11:4577 , piece_id=OJGM2TRZ6QDK4ASGD2EQKCHBWCJTR2VO3VOJMZMXJOFYHNZ5GY4A, seg_pos={0 0}, upload=133273]: Uploading piece...
Dec 10 23:37:46.966826458 (128h9m47.79453484s) [upload=133273]: Next returned nil segment
Dec 10 23:37:46.966841973 (128h9m47.794550357s) [upload=133273]: Waiting for error group managing segments...
Dec 10 23:37:46.971464654 (128h9m47.799173044s) [noise=true , address=45.91.116.6:4993 , piece_id=4WCSWAPCLQCGBDCF4CS2AFLYRANZZ3R6OPMNPMFRUMQB53SAI3GQ, seg_pos={0 0}, upload=133273]: Done uploading piece. err: <nil>
Dec 10 23:37:46.971577015 (128h9m47.799285411s) [noise=true , address=45.91.116.9:4293 , piece_id=DXMEHZM2NGXM3ITINVK455ZJ2V2G6JNQUW3BBSKOVSNW3AH66HFA, seg_pos={0 0}, upload=133273]: Done uploading piece. err: <nil>
Dec 10 23:37:46.973333602 (128h9m47.801041997s) [noise=true , address=145.79.227.48:4313 , piece_id=3YJGBWJU26CHIKQIMXGUDNL3HQESBMD5XOTPPG7FZG3UCU4JLZPQ, seg_pos={0 0}, upload=133273]: Done uploading piece. err: <nil>
Dec 10 23:37:46.973664567 (128h9m47.801372961s) [noise=true , address=145.79.227.3:4167 , piece_id=IOZDFIBKJYHMSUUI25Z4AMIHEKYMVR3UYGZRPCQ56FUHUZ662MHQ, seg_pos={0 0}, upload=133273]: Done uploading piece. err: <nil>
Dec 10 23:37:46.981484790 (128h9m47.809193186s) [noise=true , address=45.91.116.13:4180 , piece_id=M2MLX3FEUCTSIFXNMWRYRF2HH4YGDWCKKZYGTIJTOJYXPOV7D7OA, seg_pos={0 0}, upload=133273]: Done uploading piece. err: <nil>
Dec 10 23:37:46.985140038 (128h9m47.812848429s) [noise=true , address=45.91.116.5:4327 , piece_id=K3HHQ3P5NHK2VJ3MZ76O6JRY4NS4IMUMRR7UWIG7DCRBYY3GE2XA, seg_pos={0 0}, upload=133273]: Done uploading piece. err: <nil>
Dec 10 23:37:46.985426719 (128h9m47.813135104s) [noise=true , address=45.91.116.7:4022 , piece_id=CL7ZVEOG3UYPJCSMYNAHTJCUTBZX5BV6SPLZKVEBGGE5UEEBXLEQ, seg_pos={0 0}, upload=133273]: Done uploading piece. err: <nil>
Dec 10 23:37:46.985755243 (128h9m47.813463639s) [noise=true , address=45.91.116.11:4127 , piece_id=FF6Y6LNYGKTKLAQEIO5Q7XE5LBTVHIHMEORMQE5NHM6XLRCF73RA, seg_pos={0 0}, upload=133273]: Done uploading piece. err: <nil>
Dec 10 23:37:46.985989628 (128h9m47.813698019s) [noise=true , address=45.91.116.14:4204 , piece_id=EKG3TWE3KQOJ23GPEPMRGQGGXJD4S3BO2G3TM3IYVCP7WIRQSNUA, seg_pos={0 0}, upload=133273]: Done uploading piece. err: <nil>
Dec 10 23:37:47.051616827 (128h9m47.879325231s) [noise=true , address=5.231.73.4:4010 , piece_id=LZ3PZRDYY4HFFA27E5ELKBOITHDWD5XV4NHMQJPQ3FPAEC7VLJ3A, seg_pos={0 0}, upload=133273]: Done uploading piece. err: <nil>
Dec 10 23:37:47.077409129 (128h9m47.905117523s) [noise=true , address=5.231.73.3:4001 , piece_id=HCN4EP4VRVTIFKRDPIC7OBZMERZXDRUGJC2EM773WNCQX6VZRHNQ, seg_pos={0 0}, upload=133273]: Done uploading piece. err: <nil>
Dec 10 23:38:46.825144421 (128h10m47.652852833s) [noise=true , address=77.90.31.15:4705 , piece_id=DD7GJGQJ4CUFYRFGIUOACGKEYBCN6RML4XUO6DVHRVRA735JP37Q, seg_pos={0 0}, upload=133273]: Done uploading piece. err: ecclient: upload failed (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw, address:77.90.31.15:4705): protocol: rpc: dial tcp 77.90.31.15:4705: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:38:46.825491857 (128h10m47.653200258s) [noise=true , address=77.90.31.14:4673 , piece_id=ELTOW2L7HO27C3BRVJUJMSS2E5XKUUGYDPVN4CJ4O26MQV65HBHQ, seg_pos={0 0}, upload=133273]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: rpc: dial tcp 77.90.31.14:4673: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:38:46.825569094 (128h10m47.653277607s) [noise=true , address=77.90.31.21:4911 , piece_id=QW7TPOIM2FHHHHFHR6OO57IMRNPYUCIXPIRLC5BPNSFK4TCQZE3Q, seg_pos={0 0}, upload=133273]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: rpc: dial tcp 77.90.31.21:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:38:46.825837618 (128h10m47.653546021s) [noise=true , address=77.90.31.19:4869 , piece_id=SEWKXV6X2U24Q73ZW5WNAK7EIG2Z7VG3VWBVJA5KHHPVTY6UTA6Q, seg_pos={0 0}, upload=133273]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: rpc: dial tcp 77.90.31.19:4869: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:38:46.826326473 (128h10m47.654034874s) [noise=true , address=77.90.31.13:4629 , piece_id=W7EGWRNIDJH37DGGXPR6674EKAA7MFXK7IL2TDLB63E3V6DVH6OA, seg_pos={0 0}, upload=133273]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:38:46.826581384 (128h10m47.654289784s) [noise=true , address=77.90.31.11:4577 , piece_id=OJGM2TRZ6QDK4ASGD2EQKCHBWCJTR2VO3VOJMZMXJOFYHNZ5GY4A, seg_pos={0 0}, upload=133273]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: rpc: dial tcp 77.90.31.11:4577: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:38:46.826863361 (128h10m47.654571761s) [noise=true , address=77.90.31.12:4598 , piece_id=BG3NZFPB6L26IXA2NQALNWS3JHYYQDH763STNQ5OGWXLAA4L522A, seg_pos={0 0}, upload=133273]: Done uploading piece. err: ecclient: upload failed (node:12Li7UVDCVmainU4mkxtFyGky2wi6989Qu12BCQGgNn6roNX2A3, address:77.90.31.12:4598): protocol: write tcp 159.223.143.39:46948->77.90.31.12:4598: use of closed network connection; context deadline exceeded
Dec 10 23:38:46.826898284 (128h10m47.654606662s) [seg_pos={0 0}, upload=133273]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:38:46.848347664 (128h10m47.676056061s) [upload=133273]: Done uploading segments...
Dec 10 23:38:46.848386269 (128h10m47.676094663s) [upload=133273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 1m0.025093541s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......................................X....X...X........X..] duration:
475.674603ms trace:
https://jaeger.dev.storj.io/trace/2d7c696faa1ab964
FAIL (2025-12-10 23:38:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 475.696997ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.......................................XXX.X................] duration:
434.315831ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:38:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 434.359082ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
14 out of 60 is failed (threshold=3) [.............................................XXXXXXX.XXXXXXX] duration:
1m0.027475366s trace:
https://jaeger.dev.storj.io/trace/4cd791db0e698212
FAIL (2025-12-10 23:38:28 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
.895255275s) [seg_pos={0 0}, upload=133268]: Begin upload segment...
Dec 10 23:37:28.067632353 (128h9m28.895340736s) [seg_pos={0 0}, upload=133268]: Done begin upload segment.
Dec 10 23:37:28.067673654 (128h9m28.895382052s) [noise=true , address=45.91.116.8:4063 , piece_id=C2CFW5APF3QNAHOCXDPTRUSD3V2Q4JMQRI32ZDOKMCKO2CMFIU2A, seg_pos={0 0}, upload=133268]: Uploading piece...
Dec 10 23:37:28.068465885 (128h9m28.896174285s) [noise=true , address=77.90.31.10:4530 , piece_id=D3VLDBFAX2R74O2GYTL2UAVCHUIDMV4V6YASRA3Q2QUYWMBLYMRA, seg_pos={0 0}, upload=133268]: Uploading piece...
Dec 10 23:37:28.068654105 (128h9m28.896362496s) [noise=true , address=45.91.116.12:4237 , piece_id=F5MTPAM5URFWKIWYU5SZACRLS5ZFFQGBSV6BVTYCO7AXRJIQYV6A, seg_pos={0 0}, upload=133268]: Uploading piece...
Dec 10 23:37:28.069046373 (128h9m28.896754769s) [noise=true , address=5.231.73.2:4019 , piece_id=JOY3EX4U26RQKE3OQGU4PK7MR3ZTXH4Y7ZPJUYNA5IYBYIKX2WOA, seg_pos={0 0}, upload=133268]: Uploading piece...
Dec 10 23:37:28.069070605 (128h9m28.896779s) [noise=true , address=145.79.227.20:4468 , piece_id=63YPRMQOUSL6YYD4LBQOBT7LOUDJ3EO4IJBYO3GJ7XMRFLJC7K4A, seg_pos={0 0}, upload=133268]: Uploading piece...
Dec 10 23:37:28.069390789 (128h9m28.897099182s) [noise=true , address=45.91.116.9:4280 , piece_id=AZFJ54XFL4AHJG5OHNMW6VPCG5NOAL4VXK6XLPANCKXQ24CDS5VQ, seg_pos={0 0}, upload=133268]: Uploading piece...
Dec 10 23:37:28.069400958 (128h9m28.897109351s) [noise=true , address=45.91.116.6:4001 , piece_id=5A4MN27MHZB642QPZQ2KCE7C6T3SP6YCSWDQQP2B5A5I3NTECNAA, seg_pos={0 0}, upload=133268]: Uploading piece...
Dec 10 23:37:28.069723759 (128h9m28.89743216s) [noise=true , address=193.149.164.143:4699 , piece_id=VFJILXE6KIPTZAFQBVEDVQXQWS3WCQ34CQKNNF3EI37C45EKGYCA, seg_pos={0 0}, upload=133268]: Uploading piece...
Dec 10 23:37:28.069733171 (128h9m28.897441564s) [noise=true , address=77.90.31.13:4649 , piece_id=GXSF6G2KCJBF5ENA4YER2ANH4QPBWFMVSZXRUATNOBQG4TZGQHSQ, seg_pos={0 0}, upload=133268]: Uploading piece...
Dec 10 23:37:28.070086996 (128h9m28.897795393s) [noise=true , address=145.79.227.32:4867 , piece_id=4IURQNY4J4NJ7TYMTH7RPADZUPAEHTR5COFCKKGLW3WPDZ5G5WJQ, seg_pos={0 0}, upload=133268]: Uploading piece...
Dec 10 23:37:28.070117937 (128h9m28.897826328s) [noise=true , address=145.79.227.7:4323 , piece_id=OEYM6S7YUJC5OGUR6SPRC3IRETYY2BMIIZB3DH5F4NTWFNMBYHUQ, seg_pos={0 0}, upload=133268]: Uploading piece...
Dec 10 23:37:28.070418284 (128h9m28.898126676s) [noise=true , address=45.91.116.14:4213 , piece_id=ERQLX7AGYL4F4AMRGKZAF5P5W2VP5O2MUAWLVN5QVCISTDVB5WDA, seg_pos={0 0}, upload=133268]: Uploading piece...
Dec 10 23:37:28.070585441 (128h9m28.898293835s) [noise=true , address=77.90.31.19:4872 , piece_id=QUZDT47YIZCHPVPZRAE7B3Q7G53TNUWO6P7K2DQ2Q2XJQXHF5UGQ, seg_pos={0 0}, upload=133268]: Uploading piece...
Dec 10 23:37:28.070765469 (128h9m28.898473863s) [noise=true , address=77.90.31.21:4897 , piece_id=O57UJLDHGJF44X3Y3X6ZE4R4Z75LTQWBX4D4UAZUNSXP7N5MYFRA, seg_pos={0 0}, upload=133268]: Uploading piece...
Dec 10 23:37:28.071088924 (128h9m28.898797325s) [noise=true , address=77.90.31.16:4739 , piece_id=NFQWYXWIBCBEDCOEJ6ASB7MCHENLAMEXUVBAZZYFIR6KVTRDZR2A, seg_pos={0 0}, upload=133268]: Uploading piece...
Dec 10 23:37:28.071191024 (128h9m28.898899424s) [noise=true , address=77.90.31.17:4785 , piece_id=OAXANUVU47G5QWPHK3PPMHLT3TY3VJTQXZSTCZUBHP2TP7VPTCTA, seg_pos={0 0}, upload=133268]: Uploading piece...
Dec 10 23:37:28.071533153 (128h9m28.899241547s) [noise=true , address=145.79.227.44:4199 , piece_id=OKTOPP35UW5PZBKJLLY6WIX3A4DOPPEEGWU6B4WPS3DRVLVFPGCQ, seg_pos={0 0}, upload=133268]: Uploading piece...
Dec 10 23:37:28.072807533 (128h9m28.900515911s) [noise=true , address=5.231.73.5:4011 , piece_id=U4LBABTJSFSTZO473CSHMT3XGCWVLOGO253XAN7GDYA6J42DXTGA, seg_pos={0 0}, upload=133268]: Uploading piece...
Dec 10 23:37:28.138972236 (128h9m28.966680638s) [upload=133268]: Next returned nil segment
Dec 10 23:37:28.138990326 (128h9m28.966698704s) [upload=133268]: Waiting for error group managing segments...
Dec 10 23:37:28.142690439 (128h9m28.970398821s) [noise=true , address=45.91.116.14:4213 , piece_id=ERQLX7AGYL4F4AMRGKZAF5P5W2VP5O2MUAWLVN5QVCISTDVB5WDA, seg_pos={0 0}, upload=133268]: Done uploading piece. err: <nil>
Dec 10 23:37:28.142891366 (128h9m28.970599763s) [noise=true , address=45.91.116.6:4001 , piece_id=5A4MN27MHZB642QPZQ2KCE7C6T3SP6YCSWDQQP2B5A5I3NTECNAA, seg_pos={0 0}, upload=133268]: Done uploading piece. err: <nil>
Dec 10 23:37:28.145406054 (128h9m28.973114445s) [noise=true , address=145.79.227.32:4867 , piece_id=4IURQNY4J4NJ7TYMTH7RPADZUPAEHTR5COFCKKGLW3WPDZ5G5WJQ, seg_pos={0 0}, upload=133268]: Done uploading piece. err: <nil>
Dec 10 23:37:28.145635453 (128h9m28.973343847s) [noise=true , address=145.79.227.20:4468 , piece_id=63YPRMQOUSL6YYD4LBQOBT7LOUDJ3EO4IJBYO3GJ7XMRFLJC7K4A, seg_pos={0 0}, upload=133268]: Done uploading piece. err: <nil>
Dec 10 23:37:28.158786905 (128h9m28.986495286s) [noise=true , address=145.79.227.7:4323 , piece_id=OEYM6S7YUJC5OGUR6SPRC3IRETYY2BMIIZB3DH5F4NTWFNMBYHUQ, seg_pos={0 0}, upload=133268]: Done uploading piece. err: <nil>
Dec 10 23:37:28.161876025 (128h9m28.989584416s) [noise=true , address=45.91.116.8:4063 , piece_id=C2CFW5APF3QNAHOCXDPTRUSD3V2Q4JMQRI32ZDOKMCKO2CMFIU2A, seg_pos={0 0}, upload=133268]: Done uploading piece. err: <nil>
Dec 10 23:37:28.166845541 (128h9m28.994553933s) [noise=true , address=45.91.116.9:4280 , piece_id=AZFJ54XFL4AHJG5OHNMW6VPCG5NOAL4VXK6XLPANCKXQ24CDS5VQ, seg_pos={0 0}, upload=133268]: Done uploading piece. err: <nil>
Dec 10 23:37:28.230353071 (128h9m29.058061462s) [noise=true , address=5.231.73.2:4019 , piece_id=JOY3EX4U26RQKE3OQGU4PK7MR3ZTXH4Y7ZPJUYNA5IYBYIKX2WOA, seg_pos={0 0}, upload=133268]: Done uploading piece. err: <nil>
Dec 10 23:37:28.243882943 (128h9m29.071591337s) [noise=true , address=45.91.116.12:4237 , piece_id=F5MTPAM5URFWKIWYU5SZACRLS5ZFFQGBSV6BVTYCO7AXRJIQYV6A, seg_pos={0 0}, upload=133268]: Done uploading piece. err: <nil>
Dec 10 23:37:28.249111232 (128h9m29.076819632s) [noise=true , address=5.231.73.5:4011 , piece_id=U4LBABTJSFSTZO473CSHMT3XGCWVLOGO253XAN7GDYA6J42DXTGA, seg_pos={0 0}, upload=133268]: Done uploading piece. err: <nil>
Dec 10 23:37:28.765407967 (128h9m29.593116358s) [noise=true , address=193.149.164.143:4699 , piece_id=VFJILXE6KIPTZAFQBVEDVQXQWS3WCQ34CQKNNF3EI37C45EKGYCA, seg_pos={0 0}, upload=133268]: Done uploading piece. err: <nil>
Dec 10 23:37:29.043719983 (128h9m29.871428375s) [noise=true , address=145.79.227.44:4199 , piece_id=OKTOPP35UW5PZBKJLLY6WIX3A4DOPPEEGWU6B4WPS3DRVLVFPGCQ, seg_pos={0 0}, upload=133268]: Done uploading piece. err: <nil>
Dec 10 23:38:28.005925605 (128h10m28.833634002s) [noise=true , address=77.90.31.19:4872 , piece_id=QUZDT47YIZCHPVPZRAE7B3Q7G53TNUWO6P7K2DQ2Q2XJQXHF5UGQ, seg_pos={0 0}, upload=133268]: Done uploading piece. err: ecclient: upload failed (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL, address:77.90.31.19:4872): protocol: write tcp 159.223.143.39:48920->77.90.31.19:4872: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:38:28.006256942 (128h10m28.833965336s) [noise=true , address=77.90.31.21:4897 , piece_id=O57UJLDHGJF44X3Y3X6ZE4R4Z75LTQWBX4D4UAZUNSXP7N5MYFRA, seg_pos={0 0}, upload=133268]: Done uploading piece. err: ecclient: upload failed (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x, address:77.90.31.21:4897): protocol: write tcp 159.223.143.39:46178->77.90.31.21:4897: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:38:28.006539902 (128h10m28.834248296s) [noise=true , address=77.90.31.10:4530 , piece_id=D3VLDBFAX2R74O2GYTL2UAVCHUIDMV4V6YASRA3Q2QUYWMBLYMRA, seg_pos={0 0}, upload=133268]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: write tcp 159.223.143.39:33446->77.90.31.10:4530: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:38:28.006564362 (128h10m28.834272764s) [noise=true , address=77.90.31.17:4785 , piece_id=OAXANUVU47G5QWPHK3PPMHLT3TY3VJTQXZSTCZUBHP2TP7VPTCTA, seg_pos={0 0}, upload=133268]: Done uploading piece. err: ecclient: upload failed (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN, address:77.90.31.17:4785): protocol: write tcp 159.223.143.39:48594->77.90.31.17:4785: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:38:28.006840330 (128h10m28.83454873s) [noise=true , address=77.90.31.13:4649 , piece_id=GXSF6G2KCJBF5ENA4YER2ANH4QPBWFMVSZXRUATNOBQG4TZGQHSQ, seg_pos={0 0}, upload=133268]: Done uploading piece. err: ecclient: upload failed (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, address:77.90.31.13:4649): protocol: write tcp 159.223.143.39:51346->77.90.31.13:4649: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:38:28.007002803 (128h10m28.834711197s) [noise=true , address=77.90.31.16:4739 , piece_id=NFQWYXWIBCBEDCOEJ6ASB7MCHENLAMEXUVBAZZYFIR6KVTRDZR2A, seg_pos={0 0}, upload=133268]: Done uploading piece. err: ecclient: upload failed (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb, address:77.90.31.16:4739): protocol: write tcp 159.223.143.39:60014->77.90.31.16:4739: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:38:28.007055043 (128h10m28.834763441s) [seg_pos={0 0}, upload=133268]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:38:28.030638564 (128h10m28.858346956s) [upload=133268]: Done uploading segments...
Dec 10 23:38:28.030683329 (128h10m28.858391726s) [upload=133268]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1m0.027498615s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [........................................XXX.................] duration:
511.957928ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:38:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 511.986074ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [........................................XXX.X...............] duration:
457.501466ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:38:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 457.524715ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
12 out of 60 is failed (threshold=3) [.............................................XX..XXXXXX.XXXX] duration:
1m0.030307177s trace:
https://jaeger.dev.storj.io/trace/6701278f3db5d4c2
FAIL (2025-12-10 23:37:46 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
Dec 10 23:36:46.369480613 (128h8m47.197189007s) [upload=133257]: Starting upload
Dec 10 23:36:46.369579843 (128h8m47.197288236s) [upload=133257]: Done starting upload
Dec 10 23:36:46.369866185 (128h8m47.197574578s) [upload=133257]: Uploading segments...
Dec 10 23:36:46.369920358 (128h8m47.197628758s) [upload=133257]: Got next segment. Inline: false
Dec 10 23:36:46.369974440 (128h8m47.197682831s) [upload=133257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:36:46.369980417 (128h8m47.197688798s) [upload=133257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:36:46.404831471 (128h8m47.232539869s) [seg_pos={0 0}, upload=133257]: Begin upload segment...
Dec 10 23:36:46.404950221 (128h8m47.232658621s) [noise=true , address=45.91.116.10:4089 , piece_id=3F2EHFEHTBC2R5ESP63YKBW7MNYR2QEFYRJRC5WQWVJCBQ7I4WVQ, seg_pos={0 0}, upload=133257]: Uploading piece...
Dec 10 23:36:46.405796883 (128h8m47.233505283s) [noise=true , address=45.91.116.9:4264 , piece_id=TELCKTWTRBOPUNJPV2AMEUNIIXZTYO6B62IM7NMDIG4P43U6PGOA, seg_pos={0 0}, upload=133257]: Uploading piece...
Dec 10 23:36:46.406091999 (128h8m47.233800399s) [seg_pos={0 0}, upload=133257]: Done begin upload segment.
Dec 10 23:36:46.406126854 (128h8m47.233835277s) [noise=true , address=145.79.227.32:4839 , piece_id=DNSNFZKT6QNZH3TSLGIQVYWEIY5PJBVFQ73H4PXNQI2TZ76DKE7A, seg_pos={0 0}, upload=133257]: Uploading piece...
Dec 10 23:36:46.407060096 (128h8m47.234768494s) [noise=true , address=45.91.116.11:4148 , piece_id=LXB7MJSJ6HCTFHZUQURKTOY6DPCN4OU2ZYWQ3557UVNQ7CEICLEQ, seg_pos={0 0}, upload=133257]: Uploading piece...
Dec 10 23:36:46.408336013 (128h8m47.236044411s) [noise=true , address=77.90.31.19:4850 , piece_id=ON4NFHMLAJ2FOWOK2EPSAZYRUKWSGSNUVJLGV4Q64IM3NDKVMV5Q, seg_pos={0 0}, upload=133257]: Uploading piece...
Dec 10 23:36:46.408814523 (128h8m47.23652292s) [noise=true , address=77.90.31.15:4712 , piece_id=US5H2KZWGNYM3VASYGHUQTYRDBV3WBR5H5AR54EJMJKQWLW3OJAA, seg_pos={0 0}, upload=133257]: Uploading piece...
Dec 10 23:36:46.409337364 (128h8m47.237045758s) [noise=true , address=145.79.227.49:4340 , piece_id=KHKYUGQB7KI6T6MUBPR6AJ45QFBPOHZRYJBEUP5NOS73NON7KPVA, seg_pos={0 0}, upload=133257]: Uploading piece...
Dec 10 23:36:46.410271756 (128h8m47.237980153s) [noise=true , address=145.79.227.11:4756 , piece_id=RMIWKMYPTF4UA6223ETWEIMXD3PT3DSXTJ7SPMPATCSXJ3NUGIBQ, seg_pos={0 0}, upload=133257]: Uploading piece...
Dec 10 23:36:46.410757249 (128h8m47.23846565s) [noise=true , address=145.79.227.51:4427 , piece_id=QF3KKYLXMFAPCHDRHGL5E43A2IP4CMIVZEOOVXWWYRSQVBXBY5RQ, seg_pos={0 0}, upload=133257]: Uploading piece...
Dec 10 23:36:46.411143609 (128h8m47.238852003s) [noise=true , address=77.90.31.11:4567 , piece_id=RYVKNKPDWHBLUZ22KCL2ZLKA6YQD3MCFXKRUJMLEQC2VFKDEVN3Q, seg_pos={0 0}, upload=133257]: Uploading piece...
Dec 10 23:36:46.411786206 (128h8m47.239494607s) [noise=true , address=77.90.31.14:4673 , piece_id=MGIWF2R54KGQH2A3HZIQR27OYGA4XAVYMUUAF67HE5JT5SIRSE5Q, seg_pos={0 0}, upload=133257]: Uploading piece...
Dec 10 23:36:46.412509479 (128h8m47.240217873s) [noise=true , address=77.90.31.12:4608 , piece_id=GXI7VBBFAVJIY2YZRV4OIVXQ5YCYPTIM65VSLARVNNXECNJ3R7JA, seg_pos={0 0}, upload=133257]: Uploading piece...
Dec 10 23:36:46.413193557 (128h8m47.24090196s) [noise=true , address=77.90.31.17:4782 , piece_id=ZYMQMKXRKECPTV34H6M5K4OOMA4QEF5ILNMSAXGFXNVZXFWPKOFA, seg_pos={0 0}, upload=133257]: Uploading piece...
Dec 10 23:36:46.416183053 (128h8m47.243891435s) [noise=true , address=77.90.31.21:4890 , piece_id=TGW5O7GD2JWNYO2SKTIPW6BPLKOPNQHEEMXSKJA4LYSF66W67SUQ, seg_pos={0 0}, upload=133257]: Uploading piece...
Dec 10 23:36:46.418126424 (128h8m47.245834827s) [noise=true , address=45.91.116.13:4160 , piece_id=BQXTJG2FEMRNZQ3YUOO2Y72NX732FEIB733OYUFFWQ4WZQTQ4IMQ, seg_pos={0 0}, upload=133257]: Uploading piece...
Dec 10 23:36:46.419652948 (128h8m47.247361353s) [noise=true , address=145.79.227.48:4307 , piece_id=YSRX3NQBIB3RULMZLGSCV2VWKFWGCYVYQONWN7KBKXOISHXMXX6Q, seg_pos={0 0}, upload=133257]: Uploading piece...
Dec 10 23:36:46.421235158 (128h8m47.248943552s) [noise=true , address=145.79.227.21:4488 , piece_id=ZGSM6Z3DBDI2SSVPPQYGS3CH2KVDXOUHZX5757U4DXK7LLMLZGVQ, seg_pos={0 0}, upload=133257]: Uploading piece...
Dec 10 23:36:46.422948573 (128h8m47.250656977s) [noise=true , address=77.90.31.10:4541 , piece_id=KKEQUFGZB7FJVQYJBJZDZADQ7OATW7EZPBYIZHUZKUJFMFYKFHOQ, seg_pos={0 0}, upload=133257]: Uploading piece...
Dec 10 23:36:46.472691589 (128h8m47.30039997s) [upload=133257]: Next returned nil segment
Dec 10 23:36:46.472703112 (128h8m47.300411497s) [upload=133257]: Waiting for error group managing segments...
Dec 10 23:36:46.476782216 (128h8m47.304490615s) [noise=true , address=45.91.116.9:4264 , piece_id=TELCKTWTRBOPUNJPV2AMEUNIIXZTYO6B62IM7NMDIG4P43U6PGOA, seg_pos={0 0}, upload=133257]: Done uploading piece. err: <nil>
Dec 10 23:36:46.478657987 (128h8m47.306366383s) [noise=true , address=145.79.227.48:4307 , piece_id=YSRX3NQBIB3RULMZLGSCV2VWKFWGCYVYQONWN7KBKXOISHXMXX6Q, seg_pos={0 0}, upload=133257]: Done uploading piece. err: <nil>
Dec 10 23:36:46.479456148 (128h8m47.307164539s) [noise=true , address=145.79.227.51:4427 , piece_id=QF3KKYLXMFAPCHDRHGL5E43A2IP4CMIVZEOOVXWWYRSQVBXBY5RQ, seg_pos={0 0}, upload=133257]: Done uploading piece. err: <nil>
Dec 10 23:36:46.480322415 (128h8m47.308030807s) [noise=true , address=145.79.227.21:4488 , piece_id=ZGSM6Z3DBDI2SSVPPQYGS3CH2KVDXOUHZX5757U4DXK7LLMLZGVQ, seg_pos={0 0}, upload=133257]: Done uploading piece. err: <nil>
Dec 10 23:36:46.488077481 (128h8m47.315785873s) [noise=true , address=45.91.116.10:4089 , piece_id=3F2EHFEHTBC2R5ESP63YKBW7MNYR2QEFYRJRC5WQWVJCBQ7I4WVQ, seg_pos={0 0}, upload=133257]: Done uploading piece. err: <nil>
Dec 10 23:36:46.489855058 (128h8m47.317563436s) [noise=true , address=45.91.116.13:4160 , piece_id=BQXTJG2FEMRNZQ3YUOO2Y72NX732FEIB733OYUFFWQ4WZQTQ4IMQ, seg_pos={0 0}, upload=133257]: Done uploading piece. err: <nil>
Dec 10 23:36:46.493321678 (128h8m47.32103007s) [noise=true , address=145.79.227.49:4340 , piece_id=KHKYUGQB7KI6T6MUBPR6AJ45QFBPOHZRYJBEUP5NOS73NON7KPVA, seg_pos={0 0}, upload=133257]: Done uploading piece. err: <nil>
Dec 10 23:36:46.494616953 (128h8m47.322325355s) [noise=true , address=45.91.116.11:4148 , piece_id=LXB7MJSJ6HCTFHZUQURKTOY6DPCN4OU2ZYWQ3557UVNQ7CEICLEQ, seg_pos={0 0}, upload=133257]: Done uploading piece. err: <nil>
Dec 10 23:36:46.496749208 (128h8m47.324457589s) [noise=true , address=145.79.227.32:4839 , piece_id=DNSNFZKT6QNZH3TSLGIQVYWEIY5PJBVFQ73H4PXNQI2TZ76DKE7A, seg_pos={0 0}, upload=133257]: Done uploading piece. err: <nil>
Dec 10 23:36:46.498564334 (128h8m47.326272727s) [noise=true , address=145.79.227.11:4756 , piece_id=RMIWKMYPTF4UA6223ETWEIMXD3PT3DSXTJ7SPMPATCSXJ3NUGIBQ, seg_pos={0 0}, upload=133257]: Done uploading piece. err: <nil>
Dec 10 23:37:46.337482658 (128h9m47.165191055s) [noise=true , address=77.90.31.21:4890 , piece_id=TGW5O7GD2JWNYO2SKTIPW6BPLKOPNQHEEMXSKJA4LYSF66W67SUQ, seg_pos={0 0}, upload=133257]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: rpc: dial tcp 77.90.31.21:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:37:46.337507538 (128h9m47.165215935s) [noise=true , address=77.90.31.12:4608 , piece_id=GXI7VBBFAVJIY2YZRV4OIVXQ5YCYPTIM65VSLARVNNXECNJ3R7JA, seg_pos={0 0}, upload=133257]: Done uploading piece. err: ecclient: upload failed (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, address:77.90.31.12:4608): protocol: rpc: dial tcp 77.90.31.12:4608: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:37:46.337805600 (128h9m47.165513998s) [noise=true , address=77.90.31.14:4673 , piece_id=MGIWF2R54KGQH2A3HZIQR27OYGA4XAVYMUUAF67HE5JT5SIRSE5Q, seg_pos={0 0}, upload=133257]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: rpc: dial tcp 77.90.31.14:4673: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:37:46.338072625 (128h9m47.165781022s) [noise=true , address=77.90.31.19:4850 , piece_id=ON4NFHMLAJ2FOWOK2EPSAZYRUKWSGSNUVJLGV4Q64IM3NDKVMV5Q, seg_pos={0 0}, upload=133257]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:37:46.338260734 (128h9m47.165969136s) [noise=true , address=77.90.31.15:4712 , piece_id=US5H2KZWGNYM3VASYGHUQTYRDBV3WBR5H5AR54EJMJKQWLW3OJAA, seg_pos={0 0}, upload=133257]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:37:46.338291921 (128h9m47.166000321s) [noise=true , address=77.90.31.17:4782 , piece_id=ZYMQMKXRKECPTV34H6M5K4OOMA4QEF5ILNMSAXGFXNVZXFWPKOFA, seg_pos={0 0}, upload=133257]: Done uploading piece. err: ecclient: upload failed (node:12FtB4bKTHntdUQxF8G5o9nT73oS2xB7fgD44BkRfKDNFyZnKCk, address:77.90.31.17:4782): protocol: rpc: dial tcp 77.90.31.17:4782: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:37:46.338429671 (128h9m47.166138061s) [noise=true , address=77.90.31.11:4567 , piece_id=RYVKNKPDWHBLUZ22KCL2ZLKA6YQD3MCFXKRUJMLEQC2VFKDEVN3Q, seg_pos={0 0}, upload=133257]: Done uploading piece. err: ecclient: upload failed (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk, address:77.90.31.11:4567): protocol: write tcp 159.223.143.39:35444->77.90.31.11:4567: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:37:46.338646491 (128h9m47.166354888s) [noise=true , address=77.90.31.10:4541 , piece_id=KKEQUFGZB7FJVQYJBJZDZADQ7OATW7EZPBYIZHUZKUJFMFYKFHOQ, seg_pos={0 0}, upload=133257]: Done uploading piece. err: ecclient: upload failed (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW, address:77.90.31.10:4541): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:37:46.338669707 (128h9m47.166378084s) [seg_pos={0 0}, upload=133257]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:37:46.366159642 (128h9m47.193868037s) [upload=133257]: Done uploading segments...
Dec 10 23:37:46.366199163 (128h9m47.193907555s) [upload=133257]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 1m0.030321506s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [........................................X....X...X........X.] duration:
442.806778ms trace:
https://jaeger.dev.storj.io/trace/2d7c696faa1ab964
FAIL (2025-12-10 23:37:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 442.830797ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
13 out of 60 is failed (threshold=3) [..............................................XXXXXXX.XXXXXX] duration:
1m0.030046281s trace:
https://jaeger.dev.storj.io/trace/56313a652082b0d1
FAIL (2025-12-10 23:37:21 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
Item_SegmentBegin
Dec 10 23:36:21.907692529 (128h8m22.735400921s) [seg_pos={0 0}, upload=133249]: Begin upload segment...
Dec 10 23:36:21.907799915 (128h8m22.735508306s) [seg_pos={0 0}, upload=133249]: Done begin upload segment.
Dec 10 23:36:21.907818413 (128h8m22.735526815s) [noise=true , address=45.91.116.12:4241 , piece_id=NF26E5GQIX6OQTPQPHM274OSX3LR6N74S7QD764C7CV7IOI5AQNQ, seg_pos={0 0}, upload=133249]: Uploading piece...
Dec 10 23:36:21.907854081 (128h8m22.735562477s) [noise=true , address=77.90.31.21:4908 , piece_id=MJHKI3QSLTQEHYXLOV7B5G7IPIMHUIBEPELSXPPV7J3QEL4KS3ZQ, seg_pos={0 0}, upload=133249]: Uploading piece...
Dec 10 23:36:21.908483793 (128h8m22.736192184s) [noise=true , address=45.91.116.10:4113 , piece_id=DKGCBK6T2UWXABJBMXCUEXMS4JA3QZDLPMAO6O7GDQAYDORWHVIQ, seg_pos={0 0}, upload=133249]: Uploading piece...
Dec 10 23:36:21.909060693 (128h8m22.736769087s) [noise=true , address=145.79.227.15:4908 , piece_id=KL5LCTJRQU37AG7HKUA53YAUL2VBJD7ZMUWLOZANRMAOJW6MCTCQ, seg_pos={0 0}, upload=133249]: Uploading piece...
Dec 10 23:36:21.909729476 (128h8m22.737437873s) [noise=true , address=77.90.31.13:4638 , piece_id=HFOLKTVRG6MKFZKZF23S65SLQBYE2GIWHZXECXBWILQUFLTKTDDA, seg_pos={0 0}, upload=133249]: Uploading piece...
Dec 10 23:36:21.910937944 (128h8m22.738646349s) [noise=true , address=77.90.31.16:4749 , piece_id=JUCOLB2LIJJ4J6LNZYNSI5LEPWYKSHHKV7CJEZ7CNEDIRMOWF3KA, seg_pos={0 0}, upload=133249]: Uploading piece...
Dec 10 23:36:21.911679550 (128h8m22.739387931s) [noise=true , address=77.90.31.12:4605 , piece_id=C4L45AJTVTJWIH3WODTBZTZRIRVQSAU5C4UUVBI3QBF6MUJCAAKQ, seg_pos={0 0}, upload=133249]: Uploading piece...
Dec 10 23:36:21.915099249 (128h8m22.74280766s) [noise=true , address=77.90.31.11:4568 , piece_id=PC7AI34LOOOWEXR7SROIM2L4HVYPX2ITPG3FMALKTXH2GYOJR33A, seg_pos={0 0}, upload=133249]: Uploading piece...
Dec 10 23:36:21.915888103 (128h8m22.743596504s) [noise=true , address=77.90.31.21:4902 , piece_id=WEZFADEU2LBMQVXC4QTHYYLIWF5TLN54M6A6VPBZFBMUA3FY26NA, seg_pos={0 0}, upload=133249]: Uploading piece...
Dec 10 23:36:21.916587186 (128h8m22.744295592s) [noise=true , address=77.90.31.17:4773 , piece_id=ZJN3PZCBQJ72JSWTB7NNJXKB6PATHFV5I5DYJIJVS5WESST7ZKVQ, seg_pos={0 0}, upload=133249]: Uploading piece...
Dec 10 23:36:21.917273565 (128h8m22.744981956s) [noise=true , address=45.91.116.5:4325 , piece_id=D3GZXFOL7XQXSTQ5AC4L4MKHOYWOHYJVHNYRPHDJLEDC2BWLPNOQ, seg_pos={0 0}, upload=133249]: Uploading piece...
Dec 10 23:36:21.917951070 (128h8m22.745659463s) [noise=true , address=45.91.116.7:4021 , piece_id=AV3PQVJTQBKF3RQASR5RV45AQG2J57AQRFWUP5D4SZ5VYRZ2IRTA, seg_pos={0 0}, upload=133249]: Uploading piece...
Dec 10 23:36:21.918910650 (128h8m22.746619044s) [noise=true , address=77.90.31.15:4715 , piece_id=VXPPLCDRZM463IXDB233MHCDVNG2DGCH5EWV6SDKPQCXPBTFY5WQ, seg_pos={0 0}, upload=133249]: Uploading piece...
Dec 10 23:36:21.919656398 (128h8m22.747364791s) [noise=true , address=145.79.227.32:4843 , piece_id=CKAQLT7HGEEIWLZ2SVAGUMUTBYHK52MKPTYEELII4BR3UIYOCAEQ, seg_pos={0 0}, upload=133249]: Uploading piece...
Dec 10 23:36:21.921841118 (128h8m22.749549516s) [noise=true , address=45.91.116.8:4059 , piece_id=Z3W2CZADGJ2QONU33AZSBUQF7HZFY7LXO7IQ6YL7IW4ZRP5X3WRQ, seg_pos={0 0}, upload=133249]: Uploading piece...
Dec 10 23:36:21.924086043 (128h8m22.75179444s) [noise=true , address=45.91.116.6:4999 , piece_id=2MQXD6H4MDM5OTQJL5XRB4F7GDMF4UOCLRIDVYVBUROSUDLQWGVQ, seg_pos={0 0}, upload=133249]: Uploading piece...
Dec 10 23:36:21.926524664 (128h8m22.754233064s) [noise=true , address=145.79.227.29:4971 , piece_id=FJSYKQ7FTXTTD5WSKI6BCXHIO4LURFSGIPXB2C6GJULR572B5TMA, seg_pos={0 0}, upload=133249]: Uploading piece...
Dec 10 23:36:21.929203318 (128h8m22.756911712s) [noise=true , address=64.49.13.154:4929 , piece_id=SXYF25BL3JJM2KYFWSDPTFPDAJIV2KHYTLLHUK4TQL3Q6F3HJIRA, seg_pos={0 0}, upload=133249]: Uploading piece...
Dec 10 23:36:21.979946919 (128h8m22.807655312s) [upload=133249]: Next returned nil segment
Dec 10 23:36:21.979960557 (128h8m22.807668952s) [upload=133249]: Waiting for error group managing segments...
Dec 10 23:36:21.984823822 (128h8m22.812532213s) [noise=true , address=45.91.116.7:4021 , piece_id=AV3PQVJTQBKF3RQASR5RV45AQG2J57AQRFWUP5D4SZ5VYRZ2IRTA, seg_pos={0 0}, upload=133249]: Done uploading piece. err: <nil>
Dec 10 23:36:21.986080421 (128h8m22.813788816s) [noise=true , address=145.79.227.15:4908 , piece_id=KL5LCTJRQU37AG7HKUA53YAUL2VBJD7ZMUWLOZANRMAOJW6MCTCQ, seg_pos={0 0}, upload=133249]: Done uploading piece. err: <nil>
Dec 10 23:36:21.986400179 (128h8m22.814108561s) [noise=true , address=145.79.227.32:4843 , piece_id=CKAQLT7HGEEIWLZ2SVAGUMUTBYHK52MKPTYEELII4BR3UIYOCAEQ, seg_pos={0 0}, upload=133249]: Done uploading piece. err: <nil>
Dec 10 23:36:21.990184470 (128h8m22.817892861s) [noise=true , address=45.91.116.6:4999 , piece_id=2MQXD6H4MDM5OTQJL5XRB4F7GDMF4UOCLRIDVYVBUROSUDLQWGVQ, seg_pos={0 0}, upload=133249]: Done uploading piece. err: <nil>
Dec 10 23:36:21.997511590 (128h8m22.825219984s) [noise=true , address=45.91.116.8:4059 , piece_id=Z3W2CZADGJ2QONU33AZSBUQF7HZFY7LXO7IQ6YL7IW4ZRP5X3WRQ, seg_pos={0 0}, upload=133249]: Done uploading piece. err: <nil>
Dec 10 23:36:21.999706984 (128h8m22.827415374s) [noise=true , address=45.91.116.10:4113 , piece_id=DKGCBK6T2UWXABJBMXCUEXMS4JA3QZDLPMAO6O7GDQAYDORWHVIQ, seg_pos={0 0}, upload=133249]: Done uploading piece. err: <nil>
Dec 10 23:36:22.001270951 (128h8m22.828979363s) [noise=true , address=45.91.116.12:4241 , piece_id=NF26E5GQIX6OQTPQPHM274OSX3LR6N74S7QD764C7CV7IOI5AQNQ, seg_pos={0 0}, upload=133249]: Done uploading piece. err: <nil>
Dec 10 23:36:22.003222439 (128h8m22.83093083s) [noise=true , address=45.91.116.5:4325 , piece_id=D3GZXFOL7XQXSTQ5AC4L4MKHOYWOHYJVHNYRPHDJLEDC2BWLPNOQ, seg_pos={0 0}, upload=133249]: Done uploading piece. err: <nil>
Dec 10 23:36:22.003496290 (128h8m22.831204683s) [noise=true , address=145.79.227.29:4971 , piece_id=FJSYKQ7FTXTTD5WSKI6BCXHIO4LURFSGIPXB2C6GJULR572B5TMA, seg_pos={0 0}, upload=133249]: Done uploading piece. err: <nil>
Dec 10 23:36:22.584955538 (128h8m23.412663929s) [noise=true , address=64.49.13.154:4929 , piece_id=SXYF25BL3JJM2KYFWSDPTFPDAJIV2KHYTLLHUK4TQL3Q6F3HJIRA, seg_pos={0 0}, upload=133249]: Done uploading piece. err: <nil>
Dec 10 23:37:21.837184156 (128h9m22.664892546s) [noise=true , address=77.90.31.15:4715 , piece_id=VXPPLCDRZM463IXDB233MHCDVNG2DGCH5EWV6SDKPQCXPBTFY5WQ, seg_pos={0 0}, upload=133249]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 159.223.143.39:56044->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:37:21.837352521 (128h9m22.665060917s) [noise=true , address=77.90.31.13:4638 , piece_id=HFOLKTVRG6MKFZKZF23S65SLQBYE2GIWHZXECXBWILQUFLTKTDDA, seg_pos={0 0}, upload=133249]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: write tcp 159.223.143.39:44072->77.90.31.13:4638: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:37:21.838086424 (128h9m22.665794819s) [noise=true , address=77.90.31.16:4749 , piece_id=JUCOLB2LIJJ4J6LNZYNSI5LEPWYKSHHKV7CJEZ7CNEDIRMOWF3KA, seg_pos={0 0}, upload=133249]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:37:21.838257673 (128h9m22.665966073s) [noise=true , address=77.90.31.17:4773 , piece_id=ZJN3PZCBQJ72JSWTB7NNJXKB6PATHFV5I5DYJIJVS5WESST7ZKVQ, seg_pos={0 0}, upload=133249]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:37:21.838364412 (128h9m22.666072809s) [noise=true , address=77.90.31.12:4605 , piece_id=C4L45AJTVTJWIH3WODTBZTZRIRVQSAU5C4UUVBI3QBF6MUJCAAKQ, seg_pos={0 0}, upload=133249]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:37:21.838703958 (128h9m22.666412358s) [noise=true , address=77.90.31.21:4902 , piece_id=WEZFADEU2LBMQVXC4QTHYYLIWF5TLN54M6A6VPBZFBMUA3FY26NA, seg_pos={0 0}, upload=133249]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:37:21.838884099 (128h9m22.666592499s) [noise=true , address=77.90.31.11:4568 , piece_id=PC7AI34LOOOWEXR7SROIM2L4HVYPX2ITPG3FMALKTXH2GYOJR33A, seg_pos={0 0}, upload=133249]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:37:21.839230655 (128h9m22.666939056s) [noise=true , address=77.90.31.21:4908 , piece_id=MJHKI3QSLTQEHYXLOV7B5G7IPIMHUIBEPELSXPPV7J3QEL4KS3ZQ, seg_pos={0 0}, upload=133249]: Done uploading piece. err: ecclient: upload failed (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga, address:77.90.31.21:4908): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:37:21.839272396 (128h9m22.666980781s) [seg_pos={0 0}, upload=133249]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:37:21.865449846 (128h9m22.693158237s) [upload=133249]: Done uploading segments...
Dec 10 23:37:21.865484157 (128h9m22.69319255s) [upload=133249]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 1m0.030059814s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.........................................XXX................] duration:
784.408316ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:37:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 784.441814ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.........................................X....X...X........X] duration:
30.001041044s trace:
https://jaeger.dev.storj.io/trace/2d7c696faa1ab964
FAIL (2025-12-10 23:37:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001082569s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.........................................XXX.X..............] duration:
409.615371ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:36:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 409.646609ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..........................................XXX...............] duration:
557.241447ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:36:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 557.368818ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
11 out of 60 is failed (threshold=3) [..............................................XX..XXXXXX.XXX] duration:
1m0.023191555s trace:
https://jaeger.dev.storj.io/trace/5586ba3d02604eb3
FAIL (2025-12-10 23:36:33 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
Dec 10 23:35:33.461683762 (128h7m34.289392153s) [upload=133236]: Starting upload
Dec 10 23:35:33.461745812 (128h7m34.2894542s) [upload=133236]: Done starting upload
Dec 10 23:35:33.462072098 (128h7m34.289780479s) [upload=133236]: Uploading segments...
Dec 10 23:35:33.462467572 (128h7m34.290175966s) [upload=133236]: Got next segment. Inline: false
Dec 10 23:35:33.462525564 (128h7m34.290233956s) [upload=133236]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:35:33.462532088 (128h7m34.29024048s) [upload=133236]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:35:33.488022691 (128h7m34.315731086s) [seg_pos={0 0}, upload=133236]: Begin upload segment...
Dec 10 23:35:33.488082135 (128h7m34.315790527s) [seg_pos={0 0}, upload=133236]: Done begin upload segment.
Dec 10 23:35:33.488117413 (128h7m34.315825812s) [noise=true , address=45.91.116.11:4141 , piece_id=226GNFI57RERIGUZY7VUMBA3XYUUVPOGS43ZZO3JU44WJKW5IKCQ, seg_pos={0 0}, upload=133236]: Uploading piece...
Dec 10 23:35:33.489129983 (128h7m34.316838375s) [noise=true , address=77.90.31.21:4903 , piece_id=XOQ64WAZAJWKS5UMGK6WEIMSBT5HGPS3ZWLQNYEPGGCIB4H4OQMQ, seg_pos={0 0}, upload=133236]: Uploading piece...
Dec 10 23:35:33.489318385 (128h7m34.317026783s) [noise=true , address=145.79.227.3:4158 , piece_id=VFJNWRF3ZQIVJYCSRZ2UQ6PINDUFTBCPXQUI7FI7BWM424TFLLRQ, seg_pos={0 0}, upload=133236]: Uploading piece...
Dec 10 23:35:33.490572271 (128h7m34.318280667s) [noise=true , address=77.90.31.17:4798 , piece_id=BQY4733HHLENKKNKLGY2SFS43T55DGQH3NQRC2PQIPNDMI2OQMFQ, seg_pos={0 0}, upload=133236]: Uploading piece...
Dec 10 23:35:33.490849498 (128h7m34.318557895s) [noise=true , address=77.90.31.19:4853 , piece_id=4BIXXLXUFKCHCFLHO5A4VPPFFLGQWTFBW4Q3NQSQSHRQCJV36L4Q, seg_pos={0 0}, upload=133236]: Uploading piece...
Dec 10 23:35:33.491393203 (128h7m34.319101583s) [noise=true , address=45.91.116.12:4246 , piece_id=RMY25EEJM2L3EII5GHIQSFNZOGXGPINNMX5SP5TM6SVJ5P2ZMXJQ, seg_pos={0 0}, upload=133236]: Uploading piece...
Dec 10 23:35:33.491543372 (128h7m34.319251772s) [noise=true , address=145.79.227.30:4793 , piece_id=NHYQIN7ML2UFXVBYVQPVTEDH4FT5GGOTXP7MBJDMRJYUNBBL3KRQ, seg_pos={0 0}, upload=133236]: Uploading piece...
Dec 10 23:35:33.492144253 (128h7m34.319852646s) [noise=true , address=77.90.31.12:4615 , piece_id=5OELKHI527OYYA2TZQAVJOFNJBXUVVRCT33LMMAUYWJXMZ3TGV3A, seg_pos={0 0}, upload=133236]: Uploading piece...
Dec 10 23:35:33.492263288 (128h7m34.319971685s) [noise=true , address=45.91.116.14:4222 , piece_id=5Y7SYA5JO4BP7NP4RD5JHTAANTWPROXNFTLBJTPCI4OGBUGZIASQ, seg_pos={0 0}, upload=133236]: Uploading piece...
Dec 10 23:35:33.492888699 (128h7m34.320597097s) [noise=true , address=77.90.31.15:4728 , piece_id=PETPSPHUC6V4KOBFVMBZV756BWGXTFLIBPIEKPO66Y7DWWY4I2QQ, seg_pos={0 0}, upload=133236]: Uploading piece...
Dec 10 23:35:33.493001352 (128h7m34.32070975s) [noise=true , address=5.231.73.3:4025 , piece_id=SOKQBH7UGMDDSPK5SRSSKVYV53LQX5VC5UXK5SYYWBFBPHEGYQGA, seg_pos={0 0}, upload=133236]: Uploading piece...
Dec 10 23:35:33.493573287 (128h7m34.321281681s) [noise=true , address=77.90.31.16:4748 , piece_id=7J2QGD7JXFFNJDC7L4NJXC7RGLBOPCODSFUCITXQDPHVAEP66YKQ, seg_pos={0 0}, upload=133236]: Uploading piece...
Dec 10 23:35:33.493661410 (128h7m34.321369811s) [noise=true , address=145.79.227.18:4012 , piece_id=TYOITPM57EDD4MNL4SQVS5QDCUETFHIX7XY2C2BGGG2C3GRGTZFA, seg_pos={0 0}, upload=133236]: Uploading piece...
Dec 10 23:35:33.494085383 (128h7m34.321793781s) [noise=true , address=145.79.227.16:4940 , piece_id=ANLMS437HBTC7AN3VOABUYP3MTV7OD2DFN5JNV7OVP3AAEVOFSCA, seg_pos={0 0}, upload=133236]: Uploading piece...
Dec 10 23:35:33.494402072 (128h7m34.322110469s) [noise=true , address=145.79.227.45:4233 , piece_id=XIYJ26XZVLXLII7C53WO4AMGZLCUCAXA5QBWNRCC6EW7IYUR5N3A, seg_pos={0 0}, upload=133236]: Uploading piece...
Dec 10 23:35:33.494743031 (128h7m34.322451428s) [noise=true , address=45.91.116.7:4020 , piece_id=EQURQFSG4OIL3265XSGU36BZXZH5N5H2F3H2D2Q6GTFCQW755P7A, seg_pos={0 0}, upload=133236]: Uploading piece...
Dec 10 23:35:33.494967698 (128h7m34.322676079s) [noise=true , address=77.90.31.11:4564 , piece_id=RTDDB4R3CIVOIEDCWRI7FDZHCLLZEPV4MO4PMMWEDW3YPDZPNU2Q, seg_pos={0 0}, upload=133236]: Uploading piece...
Dec 10 23:35:33.495329706 (128h7m34.323038104s) [noise=true , address=77.90.31.12:4593 , piece_id=7N77ZLGIP5MSJ6KRCEM6XRA3NVKZJR244ST6VXUPZ7PYDI7MBITA, seg_pos={0 0}, upload=133236]: Uploading piece...
Dec 10 23:35:33.603118440 (128h7m34.430826821s) [upload=133236]: Next returned nil segment
Dec 10 23:35:33.603133883 (128h7m34.430842278s) [upload=133236]: Waiting for error group managing segments...
Dec 10 23:35:33.608462899 (128h7m34.436171291s) [noise=true , address=45.91.116.11:4141 , piece_id=226GNFI57RERIGUZY7VUMBA3XYUUVPOGS43ZZO3JU44WJKW5IKCQ, seg_pos={0 0}, upload=133236]: Done uploading piece. err: <nil>
Dec 10 23:35:33.612007214 (128h7m34.439715608s) [noise=true , address=145.79.227.45:4233 , piece_id=XIYJ26XZVLXLII7C53WO4AMGZLCUCAXA5QBWNRCC6EW7IYUR5N3A, seg_pos={0 0}, upload=133236]: Done uploading piece. err: <nil>
Dec 10 23:35:33.614051340 (128h7m34.441759737s) [noise=true , address=145.79.227.3:4158 , piece_id=VFJNWRF3ZQIVJYCSRZ2UQ6PINDUFTBCPXQUI7FI7BWM424TFLLRQ, seg_pos={0 0}, upload=133236]: Done uploading piece. err: <nil>
Dec 10 23:35:33.622387475 (128h7m34.450095871s) [noise=true , address=45.91.116.12:4246 , piece_id=RMY25EEJM2L3EII5GHIQSFNZOGXGPINNMX5SP5TM6SVJ5P2ZMXJQ, seg_pos={0 0}, upload=133236]: Done uploading piece. err: <nil>
Dec 10 23:35:33.625940757 (128h7m34.453649156s) [noise=true , address=145.79.227.18:4012 , piece_id=TYOITPM57EDD4MNL4SQVS5QDCUETFHIX7XY2C2BGGG2C3GRGTZFA, seg_pos={0 0}, upload=133236]: Done uploading piece. err: <nil>
Dec 10 23:35:33.626044363 (128h7m34.453752746s) [noise=true , address=45.91.116.14:4222 , piece_id=5Y7SYA5JO4BP7NP4RD5JHTAANTWPROXNFTLBJTPCI4OGBUGZIASQ, seg_pos={0 0}, upload=133236]: Done uploading piece. err: <nil>
Dec 10 23:35:33.630937926 (128h7m34.458646318s) [noise=true , address=45.91.116.7:4020 , piece_id=EQURQFSG4OIL3265XSGU36BZXZH5N5H2F3H2D2Q6GTFCQW755P7A, seg_pos={0 0}, upload=133236]: Done uploading piece. err: <nil>
Dec 10 23:35:33.672774829 (128h7m34.500483227s) [noise=true , address=5.231.73.3:4025 , piece_id=SOKQBH7UGMDDSPK5SRSSKVYV53LQX5VC5UXK5SYYWBFBPHEGYQGA, seg_pos={0 0}, upload=133236]: Done uploading piece. err: <nil>
Dec 10 23:35:34.586655456 (128h7m35.414363857s) [noise=true , address=145.79.227.30:4793 , piece_id=NHYQIN7ML2UFXVBYVQPVTEDH4FT5GGOTXP7MBJDMRJYUNBBL3KRQ, seg_pos={0 0}, upload=133236]: Done uploading piece. err: <nil>
Dec 10 23:35:34.596012229 (128h7m35.423720624s) [noise=true , address=145.79.227.16:4940 , piece_id=ANLMS437HBTC7AN3VOABUYP3MTV7OD2DFN5JNV7OVP3AAEVOFSCA, seg_pos={0 0}, upload=133236]: Done uploading piece. err: <nil>
Dec 10 23:36:33.430157017 (128h8m34.257865417s) [noise=true , address=77.90.31.17:4798 , piece_id=BQY4733HHLENKKNKLGY2SFS43T55DGQH3NQRC2PQIPNDMI2OQMFQ, seg_pos={0 0}, upload=133236]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:36:33.430368899 (128h8m34.2580773s) [noise=true , address=77.90.31.15:4728 , piece_id=PETPSPHUC6V4KOBFVMBZV756BWGXTFLIBPIEKPO66Y7DWWY4I2QQ, seg_pos={0 0}, upload=133236]: Done uploading piece. err: ecclient: upload failed (node:152bVnKGB6XVkvkjrWDsivCMxEPXq3tesh1srTzXZBcuEohFDo, address:77.90.31.15:4728): protocol: rpc: dial tcp 77.90.31.15:4728: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:36:33.430494508 (128h8m34.258202901s) [noise=true , address=77.90.31.12:4593 , piece_id=7N77ZLGIP5MSJ6KRCEM6XRA3NVKZJR244ST6VXUPZ7PYDI7MBITA, seg_pos={0 0}, upload=133236]: Done uploading piece. err: ecclient: upload failed (node:1kSxoZb9nmG6oGNytpFTsoiGh3ZPmebVVe8TZxGzANaw7EfdeF, address:77.90.31.12:4593): protocol: write tcp 159.223.143.39:44872->77.90.31.12:4593: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:36:33.430542835 (128h8m34.258251232s) [noise=true , address=77.90.31.21:4903 , piece_id=XOQ64WAZAJWKS5UMGK6WEIMSBT5HGPS3ZWLQNYEPGGCIB4H4OQMQ, seg_pos={0 0}, upload=133236]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: rpc: dial tcp 77.90.31.21:4903: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:36:33.430558854 (128h8m34.258267232s) [noise=true , address=77.90.31.16:4748 , piece_id=7J2QGD7JXFFNJDC7L4NJXC7RGLBOPCODSFUCITXQDPHVAEP66YKQ, seg_pos={0 0}, upload=133236]: Done uploading piece. err: ecclient: upload failed (node:12k122BWfbKgBxSs8MCfNFkZYh2WfZDDHLktLQRiifkPr95G7SY, address:77.90.31.16:4748): protocol: rpc: dial tcp 77.90.31.16:4748: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:36:33.430638591 (128h8m34.25834697s) [noise=true , address=77.90.31.19:4853 , piece_id=4BIXXLXUFKCHCFLHO5A4VPPFFLGQWTFBW4Q3NQSQSHRQCJV36L4Q, seg_pos={0 0}, upload=133236]: Done uploading piece. err: ecclient: upload failed (node:1WBuBdGxdPXi6VsNf4FJpn3LpWoQztM7CV6YjAW1mUDVjAUjn2, address:77.90.31.19:4853): protocol: rpc: dial tcp 77.90.31.19:4853: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:36:33.430709895 (128h8m34.258418293s) [noise=true , address=77.90.31.12:4615 , piece_id=5OELKHI527OYYA2TZQAVJOFNJBXUVVRCT33LMMAUYWJXMZ3TGV3A, seg_pos={0 0}, upload=133236]: Done uploading piece. err: ecclient: upload failed (node:1xaory9AHtW4ra15wmL2CagPKKJU2u9keFo1ntBMy8AzjpkmwR, address:77.90.31.12:4615): protocol: rpc: dial tcp 77.90.31.12:4615: i/o timeout; conn closed: op
Dec 10 23:36:33.430871049 (128h8m34.258579443s) [noise=true , address=77.90.31.11:4564 , piece_id=RTDDB4R3CIVOIEDCWRI7FDZHCLLZEPV4MO4PMMWEDW3YPDZPNU2Q, seg_pos={0 0}, upload=133236]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:42574->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:36:33.430894511 (128h8m34.258602902s) [seg_pos={0 0}, upload=133236]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:36:33.451166807 (128h8m34.278875204s) [upload=133236]: Done uploading segments...
Dec 10 23:36:33.451217338 (128h8m34.278925736s) [upload=133236]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 1m0.023208568s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..........................................XXX.X.............] duration:
470.887488ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:36:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 470.966758ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
12 out of 60 is failed (threshold=3) [...............................................XXXXXXX.XXXXX] duration:
1m0.03033642s trace:
https://jaeger.dev.storj.io/trace/daf844499c9e640
FAIL (2025-12-10 23:36:21 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ad segment...
Dec 10 23:35:21.804476522 (128h7m22.632184922s) [noise=true , address=45.91.116.10:4101 , piece_id=WBMTDWF72YEPWOMHNBW2MFZFTMRSTEBUOCXTYWMUHMR4U2QXIXWQ, seg_pos={0 0}, upload=133231]: Uploading piece...
Dec 10 23:35:21.804581125 (128h7m22.632289507s) [seg_pos={0 0}, upload=133231]: Done begin upload segment.
Dec 10 23:35:21.804606619 (128h7m22.632315001s) [noise=true , address=77.90.31.15:4728 , piece_id=PVBT3QXRM7WH2MWCD7SPDZFHUQPD6HMMEUTX75ACINEGVRWUUWNA, seg_pos={0 0}, upload=133231]: Uploading piece...
Dec 10 23:35:21.805935219 (128h7m22.633643619s) [noise=true , address=77.90.31.11:4580 , piece_id=ALYASSPQU2YY7AX7SSNH75CXQPW6XQWIM23BXHUMRWECU6N2LP3Q, seg_pos={0 0}, upload=133231]: Uploading piece...
Dec 10 23:35:21.806572477 (128h7m22.634280876s) [noise=true , address=45.91.116.9:4290 , piece_id=S5HP3SAYUZNHYCZWYMF5T7KRZXU5AGV5KLWZMCODX4RVG52SZ4PQ, seg_pos={0 0}, upload=133231]: Uploading piece...
Dec 10 23:35:21.807203910 (128h7m22.634912311s) [noise=true , address=77.90.31.17:4774 , piece_id=GBNRAILSM2CJTCV4VN7QKJHQIW2MCKGNXIOESUMLQE2QJDFPQWIA, seg_pos={0 0}, upload=133231]: Uploading piece...
Dec 10 23:35:21.807356224 (128h7m22.635064624s) [noise=true , address=77.90.31.18:4829 , piece_id=I2WIJ5BZPCAJNIWCBSLWDQB7HZRG2QHMIDPHQQYON3BKTKELJDWQ, seg_pos={0 0}, upload=133231]: Uploading piece...
Dec 10 23:35:21.807764643 (128h7m22.635473037s) [noise=true , address=77.90.31.16:4738 , piece_id=3HWVIH3VZLWBEIQDGCB4ZEQDUWHSHL5LJJTWMLCODJ7AKXBUVHTA, seg_pos={0 0}, upload=133231]: Uploading piece...
Dec 10 23:35:21.808107551 (128h7m22.635815933s) [noise=true , address=5.231.73.5:4009 , piece_id=E7J2KHK3Y2ZKI4WDNHEUJIWX6FR7U7IRUGIMEVB2V6YNRZC7YIQQ, seg_pos={0 0}, upload=133231]: Uploading piece...
Dec 10 23:35:21.808467040 (128h7m22.636175437s) [noise=true , address=45.91.116.5:4300 , piece_id=ODCOOYHSATM7YAJUSEQDWE4TYYNUMDZE3WFVIXYQ7YLLSFIG4YLQ, seg_pos={0 0}, upload=133231]: Uploading piece...
Dec 10 23:35:21.808992282 (128h7m22.636700679s) [noise=true , address=45.91.116.11:4138 , piece_id=AJLK2QBTCXQNDKBNXFNS5V43KSMBOJQ673P7VTTSSWGTO6QDZETQ, seg_pos={0 0}, upload=133231]: Uploading piece...
Dec 10 23:35:21.809310172 (128h7m22.637018551s) [noise=true , address=145.79.227.44:4220 , piece_id=BGNKLTP4ZFXPYCGNT3BDZK4POWVRR4C7J66JHNKQC3OZNEOSJVFQ, seg_pos={0 0}, upload=133231]: Uploading piece...
Dec 10 23:35:21.810121934 (128h7m22.637830331s) [noise=true , address=77.90.31.13:4650 , piece_id=RA7EZKA23XWY7D63AUD47D2KQ2QRVHJVWYXRRNCB6FUUVJKOWHEQ, seg_pos={0 0}, upload=133231]: Uploading piece...
Dec 10 23:35:21.810575751 (128h7m22.638284153s) [noise=true , address=77.90.31.19:4858 , piece_id=FVDUOGSZY7B3CV6SODLHJRE7PRH7JE3YSO5ZWCWAID7D6N5AU4YA, seg_pos={0 0}, upload=133231]: Uploading piece...
Dec 10 23:35:21.810979418 (128h7m22.638687812s) [noise=true , address=145.79.227.33:4903 , piece_id=2C6LYKRIDKPOTEAQ5X67ITPMJ3YUBHOVJOAD4XORUNTYMNFZ37RQ, seg_pos={0 0}, upload=133231]: Uploading piece...
Dec 10 23:35:21.812503953 (128h7m22.640212334s) [noise=true , address=64.49.13.169:4715 , piece_id=XDFQVQ6XBJ7HYMZEUV5MF45FXTPVLTIUMTRAOL46XVAXNNBVZLUQ, seg_pos={0 0}, upload=133231]: Uploading piece...
Dec 10 23:35:21.813755521 (128h7m22.641463918s) [noise=true , address=5.231.73.3:4034 , piece_id=IEPGCXWN7SISO6XAAOGEYQ6C2YZHSHM5IHARC2HTVOVDQWJHDMZQ, seg_pos={0 0}, upload=133231]: Uploading piece...
Dec 10 23:35:21.815503037 (128h7m22.643211439s) [noise=true , address=145.79.227.30:4774 , piece_id=3RSM4V6ICBAQXBBCQPMZ7CJHZN5FKWTEOZ2NE2ASFCOBRAAIYQ3A, seg_pos={0 0}, upload=133231]: Uploading piece...
Dec 10 23:35:21.816192392 (128h7m22.643900783s) [noise=true , address=145.79.227.45:4232 , piece_id=BWFWDLP2KCZDLZZPDRQ7VNADYPOY3OQFNXDUD2L26SS6EHWW4GHQ, seg_pos={0 0}, upload=133231]: Uploading piece...
Dec 10 23:35:21.873070941 (128h7m22.700779321s) [upload=133231]: Next returned nil segment
Dec 10 23:35:21.873087826 (128h7m22.700796217s) [upload=133231]: Waiting for error group managing segments...
Dec 10 23:35:21.876862226 (128h7m22.704570628s) [noise=true , address=45.91.116.10:4101 , piece_id=WBMTDWF72YEPWOMHNBW2MFZFTMRSTEBUOCXTYWMUHMR4U2QXIXWQ, seg_pos={0 0}, upload=133231]: Done uploading piece. err: <nil>
Dec 10 23:35:21.877467339 (128h7m22.70517573s) [noise=true , address=45.91.116.9:4290 , piece_id=S5HP3SAYUZNHYCZWYMF5T7KRZXU5AGV5KLWZMCODX4RVG52SZ4PQ, seg_pos={0 0}, upload=133231]: Done uploading piece. err: <nil>
Dec 10 23:35:21.877631134 (128h7m22.705339525s) [noise=true , address=45.91.116.5:4300 , piece_id=ODCOOYHSATM7YAJUSEQDWE4TYYNUMDZE3WFVIXYQ7YLLSFIG4YLQ, seg_pos={0 0}, upload=133231]: Done uploading piece. err: <nil>
Dec 10 23:35:21.878053927 (128h7m22.705762319s) [noise=true , address=45.91.116.11:4138 , piece_id=AJLK2QBTCXQNDKBNXFNS5V43KSMBOJQ673P7VTTSSWGTO6QDZETQ, seg_pos={0 0}, upload=133231]: Done uploading piece. err: <nil>
Dec 10 23:35:21.879053349 (128h7m22.706761744s) [noise=true , address=145.79.227.45:4232 , piece_id=BWFWDLP2KCZDLZZPDRQ7VNADYPOY3OQFNXDUD2L26SS6EHWW4GHQ, seg_pos={0 0}, upload=133231]: Done uploading piece. err: <nil>
Dec 10 23:35:21.898564787 (128h7m22.72627318s) [noise=true , address=145.79.227.33:4903 , piece_id=2C6LYKRIDKPOTEAQ5X67ITPMJ3YUBHOVJOAD4XORUNTYMNFZ37RQ, seg_pos={0 0}, upload=133231]: Done uploading piece. err: <nil>
Dec 10 23:35:21.973393281 (128h7m22.801101672s) [noise=true , address=5.231.73.3:4034 , piece_id=IEPGCXWN7SISO6XAAOGEYQ6C2YZHSHM5IHARC2HTVOVDQWJHDMZQ, seg_pos={0 0}, upload=133231]: Done uploading piece. err: <nil>
Dec 10 23:35:21.975862581 (128h7m22.803570975s) [noise=true , address=5.231.73.5:4009 , piece_id=E7J2KHK3Y2ZKI4WDNHEUJIWX6FR7U7IRUGIMEVB2V6YNRZC7YIQQ, seg_pos={0 0}, upload=133231]: Done uploading piece. err: <nil>
Dec 10 23:35:22.850403635 (128h7m23.678112033s) [noise=true , address=145.79.227.44:4220 , piece_id=BGNKLTP4ZFXPYCGNT3BDZK4POWVRR4C7J66JHNKQC3OZNEOSJVFQ, seg_pos={0 0}, upload=133231]: Done uploading piece. err: <nil>
Dec 10 23:35:22.850846801 (128h7m23.678555193s) [noise=true , address=145.79.227.30:4774 , piece_id=3RSM4V6ICBAQXBBCQPMZ7CJHZN5FKWTEOZ2NE2ASFCOBRAAIYQ3A, seg_pos={0 0}, upload=133231]: Done uploading piece. err: <nil>
Dec 10 23:35:27.791221217 (128h7m28.618929599s) [noise=true , address=64.49.13.169:4715 , piece_id=XDFQVQ6XBJ7HYMZEUV5MF45FXTPVLTIUMTRAOL46XVAXNNBVZLUQ, seg_pos={0 0}, upload=133231]: Done uploading piece. err: <nil>
Dec 10 23:36:21.732351569 (128h8m22.560059966s) [noise=true , address=77.90.31.11:4580 , piece_id=ALYASSPQU2YY7AX7SSNH75CXQPW6XQWIM23BXHUMRWECU6N2LP3Q, seg_pos={0 0}, upload=133231]: Done uploading piece. err: ecclient: upload failed (node:12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, address:77.90.31.11:4580): protocol: rpc: dial tcp 77.90.31.11:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:36:21.732392694 (128h8m22.560101089s) [noise=true , address=77.90.31.19:4858 , piece_id=FVDUOGSZY7B3CV6SODLHJRE7PRH7JE3YSO5ZWCWAID7D6N5AU4YA, seg_pos={0 0}, upload=133231]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: write tcp 159.223.143.39:43346->77.90.31.19:4858: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:36:21.732944439 (128h8m22.560652835s) [noise=true , address=77.90.31.16:4738 , piece_id=3HWVIH3VZLWBEIQDGCB4ZEQDUWHSHL5LJJTWMLCODJ7AKXBUVHTA, seg_pos={0 0}, upload=133231]: Done uploading piece. err: ecclient: upload failed (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd, address:77.90.31.16:4738): protocol: write tcp 159.223.143.39:44582->77.90.31.16:4738: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:36:21.732972222 (128h8m22.56068062s) [noise=true , address=77.90.31.17:4774 , piece_id=GBNRAILSM2CJTCV4VN7QKJHQIW2MCKGNXIOESUMLQE2QJDFPQWIA, seg_pos={0 0}, upload=133231]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: rpc: dial tcp 77.90.31.17:4774: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:36:21.733318067 (128h8m22.561026461s) [noise=true , address=77.90.31.18:4829 , piece_id=I2WIJ5BZPCAJNIWCBSLWDQB7HZRG2QHMIDPHQQYON3BKTKELJDWQ, seg_pos={0 0}, upload=133231]: Done uploading piece. err: ecclient: upload failed (node:1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, address:77.90.31.18:4829): protocol: rpc: dial tcp 77.90.31.18:4829: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:36:21.733637233 (128h8m22.56134563s) [noise=true , address=77.90.31.13:4650 , piece_id=RA7EZKA23XWY7D63AUD47D2KQ2QRVHJVWYXRRNCB6FUUVJKOWHEQ, seg_pos={0 0}, upload=133231]: Done uploading piece. err: ecclient: upload failed (node:1FNXVNAv3KeiRqTnsvQkMobyRcNFmPAAjVqSNvgs6MPJH4smVU, address:77.90.31.13:4650): protocol: rpc: dial tcp 77.90.31.13:4650: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:36:21.733833012 (128h8m22.561541409s) [noise=true , address=77.90.31.15:4728 , piece_id=PVBT3QXRM7WH2MWCD7SPDZFHUQPD6HMMEUTX75ACINEGVRWUUWNA, seg_pos={0 0}, upload=133231]: Done uploading piece. err: ecclient: upload failed (node:152bVnKGB6XVkvkjrWDsivCMxEPXq3tesh1srTzXZBcuEohFDo, address:77.90.31.15:4728): protocol: rpc: dial tcp 77.90.31.15:4728: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:36:21.733905166 (128h8m22.561613552s) [seg_pos={0 0}, upload=133231]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:36:21.759466753 (128h8m22.587175143s) [upload=133231]: Done uploading segments...
Dec 10 23:36:21.759533463 (128h8m22.58724186s) [upload=133231]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 1m0.030353971s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...........................................XXX..............] duration:
824.618687ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:36:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 824.647603ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..........................................X....X...X........] duration:
516.116106ms trace:
https://jaeger.dev.storj.io/trace/749247032308b962
FAIL (2025-12-10 23:36:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 516.632499ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
10 out of 60 is failed (threshold=3) [...............................................XX..XXXXXX.XX] duration:
1m0.030041832s trace:
https://jaeger.dev.storj.io/trace/6503209e59982c5f
FAIL (2025-12-10 23:35:29 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
Dec 10 23:34:29.980020085 (128h6m30.807728479s) [upload=133217]: Starting upload
Dec 10 23:34:29.980067837 (128h6m30.807776237s) [upload=133217]: Done starting upload
Dec 10 23:34:29.980140399 (128h6m30.807848779s) [upload=133217]: Uploading segments...
Dec 10 23:34:29.980330813 (128h6m30.808039193s) [upload=133217]: Got next segment. Inline: false
Dec 10 23:34:29.980369013 (128h6m30.808077405s) [upload=133217]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:34:29.980374913 (128h6m30.808083305s) [upload=133217]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:34:30.009531663 (128h6m30.837240057s) [seg_pos={0 0}, upload=133217]: Begin upload segment...
Dec 10 23:34:30.009605165 (128h6m30.837313556s) [seg_pos={0 0}, upload=133217]: Done begin upload segment.
Dec 10 23:34:30.009665357 (128h6m30.837373748s) [noise=true , address=77.90.31.10:4544 , piece_id=2PUHBUQ3FM6NNGKNVKCFEKEFOOMRMP2XUTS2W2WBJUH64RR2HI7Q, seg_pos={0 0}, upload=133217]: Uploading piece...
Dec 10 23:34:30.011237405 (128h6m30.838945797s) [noise=true , address=77.90.31.21:4898 , piece_id=QF5RGFYEHPDH5BUVQREKHIKZTDPT6YVVIV7SWMNHXMQ4EIURK2ZQ, seg_pos={0 0}, upload=133217]: Uploading piece...
Dec 10 23:34:30.011263391 (128h6m30.838971792s) [noise=true , address=5.231.73.2:4026 , piece_id=BNRORYKSKVUOOD6J6O43ZSRWPOAIFDPO4LP2HO3IXZQDZNISZNSQ, seg_pos={0 0}, upload=133217]: Uploading piece...
Dec 10 23:34:30.011860422 (128h6m30.839568814s) [noise=true , address=77.90.31.15:4701 , piece_id=EUYBVW6OJLNAYO4ZQYINYLG2TMG6DT63QNBURVCUV3JEB2UJQ4LA, seg_pos={0 0}, upload=133217]: Uploading piece...
Dec 10 23:34:30.011921135 (128h6m30.839629532s) [noise=true , address=77.90.31.16:4755 , piece_id=O4VMNI477K72CGIJN6K2CG5TCS2KM3XQCH6BGIZCCPGEYNBGIVPA, seg_pos={0 0}, upload=133217]: Uploading piece...
Dec 10 23:34:30.012352749 (128h6m30.840061133s) [noise=true , address=77.90.31.12:4600 , piece_id=UGILJNHPYMFS3QQYTNCY6SJ2QGNG34ZXHWS7HL2OSD2DNPM6FRYQ, seg_pos={0 0}, upload=133217]: Uploading piece...
Dec 10 23:34:30.012710316 (128h6m30.840418718s) [noise=true , address=77.90.31.11:4557 , piece_id=4OAVYT7SARHU63WQ4JC3P2JFIOX73HIN4VXUH7QHYJGLBLAH63NA, seg_pos={0 0}, upload=133217]: Uploading piece...
Dec 10 23:34:30.012914510 (128h6m30.840622888s) [noise=true , address=45.91.116.10:4115 , piece_id=EHVVJDTVNCVBGDLGXYVMV5LZJM43F4UQV3B6UWUEADX7BW53K7KA, seg_pos={0 0}, upload=133217]: Uploading piece...
Dec 10 23:34:30.013404089 (128h6m30.841112491s) [noise=true , address=145.79.227.52:4462 , piece_id=OFEFJSROWZRG2AB2UMVAAS7PYRMQJ3OGS6C65ZIX6H377O7WEQHQ, seg_pos={0 0}, upload=133217]: Uploading piece...
Dec 10 23:34:30.013689639 (128h6m30.841398041s) [noise=true , address=145.79.227.30:4786 , piece_id=2TJIVZPLYQZ5EPEAOXSC4QP7R4EWX7BMGXVPRMMVJRKATVQSQRHQ, seg_pos={0 0}, upload=133217]: Uploading piece...
Dec 10 23:34:30.013889277 (128h6m30.841597674s) [noise=true , address=5.231.73.4:4029 , piece_id=2ATMHGHDETFDSXHEGUO4XN7GTCOGAM2PBF7ITQRRW2KEFY6VDWOQ, seg_pos={0 0}, upload=133217]: Uploading piece...
Dec 10 23:34:30.014315144 (128h6m30.842023546s) [noise=true , address=145.79.227.33:4886 , piece_id=6YSXZJZ7VUYMNI6XX4VIBGRH6HXZE7OAWLIGMIAFMDPA3ZHCLPCQ, seg_pos={0 0}, upload=133217]: Uploading piece...
Dec 10 23:34:30.014384867 (128h6m30.842093259s) [noise=true , address=77.90.31.21:4900 , piece_id=GOBKSI5M3RGAE6QXZTJ45OOXCQHKRNGCWMLX3MO6MZ5K7STVLTRQ, seg_pos={0 0}, upload=133217]: Uploading piece...
Dec 10 23:34:30.014883722 (128h6m30.842592119s) [noise=true , address=77.90.31.18:4833 , piece_id=BKWS5ZFJ2RYWWWB5HZ3DNE5BGNVU2L5HICQC55JOCFOOHEFMQDEA, seg_pos={0 0}, upload=133217]: Uploading piece...
Dec 10 23:34:30.015007906 (128h6m30.84271631s) [noise=true , address=77.90.31.19:4866 , piece_id=5TQOOUQX5MLQTBFWJDKQIAJKHHHMP55Q5FP4CDMMDIZE7RMWZBQA, seg_pos={0 0}, upload=133217]: Uploading piece...
Dec 10 23:34:30.015391302 (128h6m30.8430997s) [noise=true , address=64.49.13.26:4955 , piece_id=TA4HHCXENHQUTIEYTXUJDWRLBPZ4V22OBMNAO6HOOUDBKZ4DXPAA, seg_pos={0 0}, upload=133217]: Uploading piece...
Dec 10 23:34:30.015671521 (128h6m30.843379916s) [noise=true , address=77.90.31.12:4614 , piece_id=QVQTOSEKFGPFWK6NOTI4B3N2KO33VNNYHO4AGFZYHWGE4UGL3P4Q, seg_pos={0 0}, upload=133217]: Uploading piece...
Dec 10 23:34:30.016304861 (128h6m30.844013254s) [noise=true , address=45.91.116.11:4139 , piece_id=7XHBBEMMXX3BSY4VKY5IPIIBQHYN7AG6MXFYK66N4FD6CYOSEIRQ, seg_pos={0 0}, upload=133217]: Uploading piece...
Dec 10 23:34:30.081762949 (128h6m30.90947134s) [upload=133217]: Next returned nil segment
Dec 10 23:34:30.081779366 (128h6m30.909487761s) [upload=133217]: Waiting for error group managing segments...
Dec 10 23:34:30.088989101 (128h6m30.916697499s) [noise=true , address=145.79.227.33:4886 , piece_id=6YSXZJZ7VUYMNI6XX4VIBGRH6HXZE7OAWLIGMIAFMDPA3ZHCLPCQ, seg_pos={0 0}, upload=133217]: Done uploading piece. err: <nil>
Dec 10 23:34:30.089767058 (128h6m30.91747546s) [noise=true , address=145.79.227.52:4462 , piece_id=OFEFJSROWZRG2AB2UMVAAS7PYRMQJ3OGS6C65ZIX6H377O7WEQHQ, seg_pos={0 0}, upload=133217]: Done uploading piece. err: <nil>
Dec 10 23:34:30.099375139 (128h6m30.927083529s) [noise=true , address=45.91.116.10:4115 , piece_id=EHVVJDTVNCVBGDLGXYVMV5LZJM43F4UQV3B6UWUEADX7BW53K7KA, seg_pos={0 0}, upload=133217]: Done uploading piece. err: <nil>
Dec 10 23:34:30.109864358 (128h6m30.937572753s) [noise=true , address=45.91.116.11:4139 , piece_id=7XHBBEMMXX3BSY4VKY5IPIIBQHYN7AG6MXFYK66N4FD6CYOSEIRQ, seg_pos={0 0}, upload=133217]: Done uploading piece. err: <nil>
Dec 10 23:34:30.198452563 (128h6m31.026160954s) [noise=true , address=5.231.73.4:4029 , piece_id=2ATMHGHDETFDSXHEGUO4XN7GTCOGAM2PBF7ITQRRW2KEFY6VDWOQ, seg_pos={0 0}, upload=133217]: Done uploading piece. err: <nil>
Dec 10 23:34:30.223817940 (128h6m31.051526331s) [noise=true , address=5.231.73.2:4026 , piece_id=BNRORYKSKVUOOD6J6O43ZSRWPOAIFDPO4LP2HO3IXZQDZNISZNSQ, seg_pos={0 0}, upload=133217]: Done uploading piece. err: <nil>
Dec 10 23:34:30.608322701 (128h6m31.436031093s) [noise=true , address=64.49.13.26:4955 , piece_id=TA4HHCXENHQUTIEYTXUJDWRLBPZ4V22OBMNAO6HOOUDBKZ4DXPAA, seg_pos={0 0}, upload=133217]: Done uploading piece. err: <nil>
Dec 10 23:34:31.042580393 (128h6m31.870288785s) [noise=true , address=145.79.227.30:4786 , piece_id=2TJIVZPLYQZ5EPEAOXSC4QP7R4EWX7BMGXVPRMMVJRKATVQSQRHQ, seg_pos={0 0}, upload=133217]: Done uploading piece. err: <nil>
Dec 10 23:35:29.941415552 (128h7m30.76912395s) [noise=true , address=77.90.31.15:4701 , piece_id=EUYBVW6OJLNAYO4ZQYINYLG2TMG6DT63QNBURVCUV3JEB2UJQ4LA, seg_pos={0 0}, upload=133217]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 159.223.143.39:59180->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:35:29.941627884 (128h7m30.769336284s) [noise=true , address=77.90.31.21:4900 , piece_id=GOBKSI5M3RGAE6QXZTJ45OOXCQHKRNGCWMLX3MO6MZ5K7STVLTRQ, seg_pos={0 0}, upload=133217]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 159.223.143.39:60978->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:35:29.941836713 (128h7m30.76954511s) [noise=true , address=77.90.31.12:4614 , piece_id=QVQTOSEKFGPFWK6NOTI4B3N2KO33VNNYHO4AGFZYHWGE4UGL3P4Q, seg_pos={0 0}, upload=133217]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 159.223.143.39:33428->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:35:29.942017925 (128h7m30.769726318s) [noise=true , address=77.90.31.10:4544 , piece_id=2PUHBUQ3FM6NNGKNVKCFEKEFOOMRMP2XUTS2W2WBJUH64RR2HI7Q, seg_pos={0 0}, upload=133217]: Done uploading piece. err: ecclient: upload failed (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE, address:77.90.31.10:4544): protocol: write tcp 159.223.143.39:44366->77.90.31.10:4544: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:35:29.942177232 (128h7m30.769885627s) [noise=true , address=77.90.31.21:4898 , piece_id=QF5RGFYEHPDH5BUVQREKHIKZTDPT6YVVIV7SWMNHXMQ4EIURK2ZQ, seg_pos={0 0}, upload=133217]: Done uploading piece. err: ecclient: upload failed (node:12Qi796rSqngMJNBiFji2g4eYRn5MsqCqhpGPdYPN2GN8yFGTUx, address:77.90.31.21:4898): protocol: write tcp 159.223.143.39:45302->77.90.31.21:4898: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:35:29.943130789 (128h7m30.770839191s) [noise=true , address=77.90.31.16:4755 , piece_id=O4VMNI477K72CGIJN6K2CG5TCS2KM3XQCH6BGIZCCPGEYNBGIVPA, seg_pos={0 0}, upload=133217]: Done uploading piece. err: ecclient: upload failed (node:1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, address:77.90.31.16:4755): protocol: write tcp 159.223.143.39:46582->77.90.31.16:4755: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:35:29.943341069 (128h7m30.771049467s) [noise=true , address=77.90.31.19:4866 , piece_id=5TQOOUQX5MLQTBFWJDKQIAJKHHHMP55Q5FP4CDMMDIZE7RMWZBQA, seg_pos={0 0}, upload=133217]: Done uploading piece. err: ecclient: upload failed (node:12uCbmPHVXpukjQahs2G6oh7VY2zDMooMAbdgSyHN7Biuesmz3A, address:77.90.31.19:4866): protocol: write tcp 159.223.143.39:58562->77.90.31.19:4866: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:35:29.943429654 (128h7m30.771138054s) [noise=true , address=77.90.31.12:4600 , piece_id=UGILJNHPYMFS3QQYTNCY6SJ2QGNG34ZXHWS7HL2OSD2DNPM6FRYQ, seg_pos={0 0}, upload=133217]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: write tcp 159.223.143.39:45170->77.90.31.12:4600: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:35:29.943580752 (128h7m30.771289146s) [noise=true , address=77.90.31.11:4557 , piece_id=4OAVYT7SARHU63WQ4JC3P2JFIOX73HIN4VXUH7QHYJGLBLAH63NA, seg_pos={0 0}, upload=133217]: Done uploading piece. err: ecclient: upload failed (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX, address:77.90.31.11:4557): protocol: write tcp 159.223.143.39:43624->77.90.31.11:4557: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:35:29.943638635 (128h7m30.771347036s) [noise=true , address=77.90.31.18:4833 , piece_id=BKWS5ZFJ2RYWWWB5HZ3DNE5BGNVU2L5HICQC55JOCFOOHEFMQDEA, seg_pos={0 0}, upload=133217]: Done uploading piece. err: ecclient: upload failed (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, address:77.90.31.18:4833): protocol: write tcp 159.223.143.39:54554->77.90.31.18:4833: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:35:29.943681216 (128h7m30.771389609s) [seg_pos={0 0}, upload=133217]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Dec 10 23:35:29.968742990 (128h7m30.796451382s) [upload=133217]: Done uploading segments...
Dec 10 23:35:29.968791745 (128h7m30.796500137s) [upload=133217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 1m0.030074177s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [............................................XXX.............] duration:
549.47103ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:35:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 549.497993ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...........................................X....X...X.......] duration:
492.053769ms trace:
https://jaeger.dev.storj.io/trace/749247032308b962
FAIL (2025-12-10 23:35:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 492.082859ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
11 out of 60 is failed (threshold=3) [................................................XXXXXXX.XXXX] duration:
1m0.02508433s trace:
https://jaeger.dev.storj.io/trace/34963884d87b9921
FAIL (2025-12-10 23:35:21 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
0.31.21:4888 , piece_id=OSZCMLSMCKU554APF25YXMYIDFQBL4WJIZETGYJBKYZQ4JHDCGYQ, seg_pos={0 0}, upload=133215]: Uploading piece...
Dec 10 23:34:21.718432780 (128h6m22.546141177s) [noise=true , address=45.91.116.7:4029 , piece_id=44CL6MDT6MXG7IWVHL5GNED7BSMQRQQUGFD4NS735LEAUPHKZGJA, seg_pos={0 0}, upload=133215]: Uploading piece...
Dec 10 23:34:21.718726207 (128h6m22.546434599s) [noise=true , address=45.91.116.4:4951 , piece_id=S5HMUZHKJXL2HY2QLBYG3SRZYCOYEQ2TYQSYA2VYWXRZUDOSQS6Q, seg_pos={0 0}, upload=133215]: Uploading piece...
Dec 10 23:34:21.719196960 (128h6m22.546905358s) [noise=true , address=145.79.227.23:4558 , piece_id=5PEX3VS7FP6HAVIRXEYGGFE5BN5FWQGJ6WCENG4SXIM35TYQKKXQ, seg_pos={0 0}, upload=133215]: Uploading piece...
Dec 10 23:34:21.719252928 (128h6m22.546961319s) [noise=true , address=64.49.13.169:4707 , piece_id=7LGRK5BBIJXBUNNU4UI5C4DGALVNMKWQCC6UJR25ZSNBRIZBXW7Q, seg_pos={0 0}, upload=133215]: Uploading piece...
Dec 10 23:34:21.719736466 (128h6m22.547444865s) [noise=true , address=77.90.31.11:4563 , piece_id=52OXYNCUUKFBSFO26EHPJO625XFBHU2CPEFIIL3YDWWKQQS5VWOQ, seg_pos={0 0}, upload=133215]: Uploading piece...
Dec 10 23:34:21.719859070 (128h6m22.547567461s) [noise=true , address=45.91.116.9:4264 , piece_id=SX7XZ6YS5ZKZ2MIDYGZIRX3OBIRYLOJY5SY6D3RHRHQWBDXQLXGA, seg_pos={0 0}, upload=133215]: Uploading piece...
Dec 10 23:34:21.720211256 (128h6m22.54791965s) [noise=true , address=145.79.227.26:4660 , piece_id=JV5KCDV7ENMOCKBTNEJRFGHUWUUXPRLJYUH7SCUDQBHDIMV6MFJQ, seg_pos={0 0}, upload=133215]: Uploading piece...
Dec 10 23:34:21.720459051 (128h6m22.548167442s) [noise=true , address=77.90.31.14:4691 , piece_id=PXE2KCNEEDI25BYHPDHHPAPGNWDID66AZ2FJFPTOB3SB3GLALP2Q, seg_pos={0 0}, upload=133215]: Uploading piece...
Dec 10 23:34:21.720774157 (128h6m22.548482548s) [noise=true , address=77.90.31.13:4640 , piece_id=A6QLH3TIYNQA6YYEM4F44662PROCEC5IZ42K6AKXEISXJ3H3TZQQ, seg_pos={0 0}, upload=133215]: Uploading piece...
Dec 10 23:34:21.721678491 (128h6m22.549386889s) [noise=true , address=77.90.31.15:4718 , piece_id=X3DDGY6XQL5RTUKKELYVAZLZ3JD7DJW2GFJMVHFFVXRU5KQGCQBA, seg_pos={0 0}, upload=133215]: Uploading piece...
Dec 10 23:34:21.721947406 (128h6m22.549655802s) [noise=true , address=45.91.116.12:4229 , piece_id=GJX4NNEXIGCGO2BZNHGCZWOJUIXAXF6BLXKAFUSZ7C7CLEJKWKJA, seg_pos={0 0}, upload=133215]: Uploading piece...
Dec 10 23:34:21.723087583 (128h6m22.550795978s) [noise=true , address=64.49.13.170:4723 , piece_id=JUR47OJ6X5VP5M5M5RA3BIEYAR27DJWTW6ISP4BBLQ6I7IYVXRZA, seg_pos={0 0}, upload=133215]: Uploading piece...
Dec 10 23:34:21.724816778 (128h6m22.552525169s) [noise=true , address=45.91.116.10:4097 , piece_id=VM37B5D3OJJMBW6V55466FQXFTL6WE6F4YSVGTR6346AJ3WYG72A, seg_pos={0 0}, upload=133215]: Uploading piece...
Dec 10 23:34:21.726258025 (128h6m22.553966429s) [noise=true , address=77.90.31.18:4830 , piece_id=KHUQPVZA2DUZ776U44TCTDC2ZFCW6K5MJOUV3DVUQUHSQGT6UEXQ, seg_pos={0 0}, upload=133215]: Uploading piece...
Dec 10 23:34:21.727678475 (128h6m22.55538692s) [noise=true , address=145.79.227.42:4150 , piece_id=YHFSBHTSXZAPDJAXS3ORZLC7J4VQ5IYQO5HLFQO5F5JD2CUCO7BQ, seg_pos={0 0}, upload=133215]: Uploading piece...
Dec 10 23:34:21.728175470 (128h6m22.555883867s) [noise=true , address=77.90.31.17:4783 , piece_id=KESRYJFGRHNSPAKCFRBP57P5LSNE545OV6B5HPXVKPHKYYGSEY3A, seg_pos={0 0}, upload=133215]: Uploading piece...
Dec 10 23:34:21.729079150 (128h6m22.556787545s) [noise=true , address=145.79.227.47:4286 , piece_id=42RL423ZCXF5AEY3D2BTW34F2BNP6XXLIJZADZTSLOPU3NXJGDCQ, seg_pos={0 0}, upload=133215]: Uploading piece...
Dec 10 23:34:21.786380467 (128h6m22.614088849s) [upload=133215]: Next returned nil segment
Dec 10 23:34:21.786393426 (128h6m22.614101818s) [upload=133215]: Waiting for error group managing segments...
Dec 10 23:34:21.790212776 (128h6m22.617921168s) [noise=true , address=45.91.116.4:4951 , piece_id=S5HMUZHKJXL2HY2QLBYG3SRZYCOYEQ2TYQSYA2VYWXRZUDOSQS6Q, seg_pos={0 0}, upload=133215]: Done uploading piece. err: <nil>
Dec 10 23:34:21.790301818 (128h6m22.618010209s) [noise=true , address=45.91.116.10:4097 , piece_id=VM37B5D3OJJMBW6V55466FQXFTL6WE6F4YSVGTR6346AJ3WYG72A, seg_pos={0 0}, upload=133215]: Done uploading piece. err: <nil>
Dec 10 23:34:21.793277213 (128h6m22.620985613s) [noise=true , address=145.79.227.23:4558 , piece_id=5PEX3VS7FP6HAVIRXEYGGFE5BN5FWQGJ6WCENG4SXIM35TYQKKXQ, seg_pos={0 0}, upload=133215]: Done uploading piece. err: <nil>
Dec 10 23:34:21.794216742 (128h6m22.621925133s) [noise=true , address=145.79.227.26:4660 , piece_id=JV5KCDV7ENMOCKBTNEJRFGHUWUUXPRLJYUH7SCUDQBHDIMV6MFJQ, seg_pos={0 0}, upload=133215]: Done uploading piece. err: <nil>
Dec 10 23:34:21.797225390 (128h6m22.624933782s) [noise=true , address=145.79.227.47:4286 , piece_id=42RL423ZCXF5AEY3D2BTW34F2BNP6XXLIJZADZTSLOPU3NXJGDCQ, seg_pos={0 0}, upload=133215]: Done uploading piece. err: <nil>
Dec 10 23:34:21.801649862 (128h6m22.629358252s) [noise=true , address=45.91.116.9:4264 , piece_id=SX7XZ6YS5ZKZ2MIDYGZIRX3OBIRYLOJY5SY6D3RHRHQWBDXQLXGA, seg_pos={0 0}, upload=133215]: Done uploading piece. err: <nil>
Dec 10 23:34:21.807684939 (128h6m22.63539333s) [noise=true , address=45.91.116.12:4229 , piece_id=GJX4NNEXIGCGO2BZNHGCZWOJUIXAXF6BLXKAFUSZ7C7CLEJKWKJA, seg_pos={0 0}, upload=133215]: Done uploading piece. err: <nil>
Dec 10 23:34:21.808677953 (128h6m22.636386335s) [noise=true , address=45.91.116.7:4029 , piece_id=44CL6MDT6MXG7IWVHL5GNED7BSMQRQQUGFD4NS735LEAUPHKZGJA, seg_pos={0 0}, upload=133215]: Done uploading piece. err: <nil>
Dec 10 23:34:21.809957049 (128h6m22.637665444s) [noise=true , address=145.79.227.42:4150 , piece_id=YHFSBHTSXZAPDJAXS3ORZLC7J4VQ5IYQO5HLFQO5F5JD2CUCO7BQ, seg_pos={0 0}, upload=133215]: Done uploading piece. err: <nil>
Dec 10 23:34:22.376933402 (128h6m23.204641794s) [noise=true , address=64.49.13.170:4723 , piece_id=JUR47OJ6X5VP5M5M5RA3BIEYAR27DJWTW6ISP4BBLQ6I7IYVXRZA, seg_pos={0 0}, upload=133215]: Done uploading piece. err: <nil>
Dec 10 23:34:27.769010688 (128h6m28.596719082s) [noise=true , address=64.49.13.169:4707 , piece_id=7LGRK5BBIJXBUNNU4UI5C4DGALVNMKWQCC6UJR25ZSNBRIZBXW7Q, seg_pos={0 0}, upload=133215]: Done uploading piece. err: <nil>
Dec 10 23:35:21.647965311 (128h7m22.475673708s) [noise=true , address=77.90.31.17:4783 , piece_id=KESRYJFGRHNSPAKCFRBP57P5LSNE545OV6B5HPXVKPHKYYGSEY3A, seg_pos={0 0}, upload=133215]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 159.223.143.39:41464->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:35:21.648167907 (128h7m22.475876304s) [noise=true , address=77.90.31.21:4888 , piece_id=OSZCMLSMCKU554APF25YXMYIDFQBL4WJIZETGYJBKYZQ4JHDCGYQ, seg_pos={0 0}, upload=133215]: Done uploading piece. err: ecclient: upload failed (node:12BiqwcDuvobZabx3jxDhXf27Ei8oLFS6wLp3qbpZCtJ4ZhQEZA, address:77.90.31.21:4888): protocol: write tcp 159.223.143.39:60500->77.90.31.21:4888: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:35:21.648281005 (128h7m22.475989396s) [noise=true , address=77.90.31.18:4830 , piece_id=KHUQPVZA2DUZ776U44TCTDC2ZFCW6K5MJOUV3DVUQUHSQGT6UEXQ, seg_pos={0 0}, upload=133215]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 159.223.143.39:50478->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:35:21.648498118 (128h7m22.47620651s) [noise=true , address=77.90.31.11:4563 , piece_id=52OXYNCUUKFBSFO26EHPJO625XFBHU2CPEFIIL3YDWWKQQS5VWOQ, seg_pos={0 0}, upload=133215]: Done uploading piece. err: ecclient: upload failed (node:12bCw4Dx7aFbxT7TeUKezRzHD4yQWYXXNJVgFfF3FGhnkVVk9Jb, address:77.90.31.11:4563): protocol: write tcp 159.223.143.39:35526->77.90.31.11:4563: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:35:21.648688227 (128h7m22.476396628s) [noise=true , address=77.90.31.14:4691 , piece_id=PXE2KCNEEDI25BYHPDHHPAPGNWDID66AZ2FJFPTOB3SB3GLALP2Q, seg_pos={0 0}, upload=133215]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: write tcp 159.223.143.39:47782->77.90.31.14:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:35:21.648865486 (128h7m22.47657388s) [noise=true , address=77.90.31.15:4718 , piece_id=X3DDGY6XQL5RTUKKELYVAZLZ3JD7DJW2GFJMVHFFVXRU5KQGCQBA, seg_pos={0 0}, upload=133215]: Done uploading piece. err: ecclient: upload failed (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC, address:77.90.31.15:4718): protocol: write tcp 159.223.143.39:46452->77.90.31.15:4718: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:35:21.648891094 (128h7m22.476599488s) [noise=true , address=77.90.31.13:4640 , piece_id=A6QLH3TIYNQA6YYEM4F44662PROCEC5IZ42K6AKXEISXJ3H3TZQQ, seg_pos={0 0}, upload=133215]: Done uploading piece. err: ecclient: upload failed (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F, address:77.90.31.13:4640): protocol: write tcp 159.223.143.39:48798->77.90.31.13:4640: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:35:21.648918903 (128h7m22.47662729s) [seg_pos={0 0}, upload=133215]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:35:21.671225338 (128h7m22.498933728s) [upload=133215]: Done uploading segments...
Dec 10 23:35:21.671279472 (128h7m22.498987872s) [upload=133215]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 1m0.025112193s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...........................................XXX.X............] duration:
462.666392ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:35:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 462.706646ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
9 out of 60 is failed (threshold=3) [................................................XX..XXXXXX.X] duration:
1m0.026994191s trace:
https://jaeger.dev.storj.io/trace/c5e18cfa15c2f5f
FAIL (2025-12-10 23:34:27 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Dec 10 23:33:27.969203581 (128h5m28.796911982s) [upload=133203]: Starting upload
Dec 10 23:33:27.969296555 (128h5m28.797004953s) [upload=133203]: Done starting upload
Dec 10 23:33:27.969503935 (128h5m28.797212315s) [upload=133203]: Uploading segments...
Dec 10 23:33:27.969983211 (128h5m28.797691608s) [upload=133203]: Got next segment. Inline: false
Dec 10 23:33:27.970036303 (128h5m28.797744694s) [upload=133203]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:33:27.970068810 (128h5m28.797777192s) [upload=133203]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:33:27.995709244 (128h5m28.823417627s) [seg_pos={0 0}, upload=133203]: Begin upload segment...
Dec 10 23:33:27.995789762 (128h5m28.823498142s) [seg_pos={0 0}, upload=133203]: Done begin upload segment.
Dec 10 23:33:27.995823726 (128h5m28.823532124s) [noise=true , address=77.90.31.13:4635 , piece_id=D762FEUXE7HCL26VSIUV2VRNDGVDBY7SKQPC6GLHHL4YBZWV6H5A, seg_pos={0 0}, upload=133203]: Uploading piece...
Dec 10 23:33:27.996855096 (128h5m28.824563498s) [noise=true , address=145.79.227.39:4021 , piece_id=PNWLJJV5PXQLBD4HV6SW4UGQF4VELIAFAIQMRR3NGQIJG3M54FTA, seg_pos={0 0}, upload=133203]: Uploading piece...
Dec 10 23:33:27.997385925 (128h5m28.825094327s) [noise=true , address=145.79.227.25:4647 , piece_id=BQZLQ5UEV6VOU6UVU3FZJ3FKPOVAOZQKD2EOK5O63EH4FEDNIEEA, seg_pos={0 0}, upload=133203]: Uploading piece...
Dec 10 23:33:27.997825209 (128h5m28.825533611s) [noise=true , address=145.79.227.3:4163 , piece_id=JWX5TES3CBKKV676VS2Q44AAY7RYJLWHMLMHPGOD3CPHSUW3CZCA, seg_pos={0 0}, upload=133203]: Uploading piece...
Dec 10 23:33:27.998267476 (128h5m28.825975879s) [noise=true , address=77.90.31.15:4707 , piece_id=7RAN7KV5KJLV2RGO4SOHANSJB32IYHEAWLY5737DDFTDY6KCTYUA, seg_pos={0 0}, upload=133203]: Uploading piece...
Dec 10 23:33:27.998715910 (128h5m28.826424312s) [noise=true , address=45.91.116.10:4087 , piece_id=DBID6UJRTQPMNCQVZIKUUKHOJMQPICPRN6NUREDI3LW4HKJ6HMKA, seg_pos={0 0}, upload=133203]: Uploading piece...
Dec 10 23:33:27.999223186 (128h5m28.826931588s) [noise=true , address=145.79.227.22:4515 , piece_id=LXMGXZAGC6JNM7P7C67ZIDMJ2TRH52YT7OSKIHVN3ROU46L6CZPQ, seg_pos={0 0}, upload=133203]: Uploading piece...
Dec 10 23:33:27.999812367 (128h5m28.827520777s) [noise=true , address=77.90.31.12:4608 , piece_id=JHJYWI3G53BYKDQMTL4T5EBN6NIXI5HI4OMXZNAQA5OXA5DEAGHQ, seg_pos={0 0}, upload=133203]: Uploading piece...
Dec 10 23:33:28.000362082 (128h5m28.828070484s) [noise=true , address=145.79.227.38:4004 , piece_id=VGBBN4MUBMAZANLXKGEUD6HHPWNEXFXA2TNJBCGUCEWJ5BXTY5IQ, seg_pos={0 0}, upload=133203]: Uploading piece...
Dec 10 23:33:28.000847944 (128h5m28.828556347s) [noise=true , address=77.90.31.16:4764 , piece_id=EXZO4JXQL6WFI2ENZE2ZIKR5U2VQD7GGOQQM57ETGPI5VIO5ZLQQ, seg_pos={0 0}, upload=133203]: Uploading piece...
Dec 10 23:33:28.001307549 (128h5m28.829015951s) [noise=true , address=77.90.31.19:4870 , piece_id=K4WCAKT6KYIL3AMNJSUG7IAROEDE5AP75PV3BBBWKV5DQ2IIEFQQ, seg_pos={0 0}, upload=133203]: Uploading piece...
Dec 10 23:33:28.001916114 (128h5m28.829624556s) [noise=true , address=64.49.13.169:4709 , piece_id=B3A2ROTXNODVCTJMAMO5UGNBFJCTU4UBHD6NIHH6IVPGCM2VPSAA, seg_pos={0 0}, upload=133203]: Uploading piece...
Dec 10 23:33:28.002412218 (128h5m28.830120619s) [noise=true , address=145.79.227.22:4519 , piece_id=KUANZVO4BRLZWH47PKEHA2DYHZXZAF5YSBPARY3VNCC3N24B35JA, seg_pos={0 0}, upload=133203]: Uploading piece...
Dec 10 23:33:28.002887098 (128h5m28.8305955s) [noise=true , address=45.91.116.11:4125 , piece_id=EE6IHZLI7QBKAF7KB4Y6BBQLJI5HQU42HRGMWN4PQMXGJ2G5U5NQ, seg_pos={0 0}, upload=133203]: Uploading piece...
Dec 10 23:33:28.003490287 (128h5m28.831198689s) [noise=true , address=5.231.73.4:4010 , piece_id=CTNYKYALWPEA7OMXZ5GSNAWW2C7WYL3GVQKERPZ6INPV44Y2LD7A, seg_pos={0 0}, upload=133203]: Uploading piece...
Dec 10 23:33:28.004019068 (128h5m28.83172747s) [noise=true , address=45.91.116.8:4072 , piece_id=WC6W6J6GKYYOGHAKIYQVWYAMBIFR2NR5PO4VM2ZNRDH535BPGVUA, seg_pos={0 0}, upload=133203]: Uploading piece...
Dec 10 23:33:28.004717174 (128h5m28.832425616s) [noise=true , address=77.90.31.14:4673 , piece_id=AIVDB3XODE4FPD6SNWLGCKEXRI5JW62XKVMFL3QZDLT7GD4QWEVA, seg_pos={0 0}, upload=133203]: Uploading piece...
Dec 10 23:33:28.006824982 (128h5m28.834533383s) [noise=true , address=45.91.116.11:4138 , piece_id=V7SNI6FAPS4ZKH7KI32I7PJZWQHWJVUC5B6KFJGEZ7FV7IRXQZGQ, seg_pos={0 0}, upload=133203]: Uploading piece...
Dec 10 23:33:28.077448762 (128h5m28.905157154s) [upload=133203]: Next returned nil segment
Dec 10 23:33:28.077466186 (128h5m28.905174591s) [upload=133203]: Waiting for error group managing segments...
Dec 10 23:33:28.083655930 (128h5m28.91136431s) [noise=true , address=145.79.227.22:4519 , piece_id=KUANZVO4BRLZWH47PKEHA2DYHZXZAF5YSBPARY3VNCC3N24B35JA, seg_pos={0 0}, upload=133203]: Done uploading piece. err: <nil>
Dec 10 23:33:28.083793363 (128h5m28.911501773s) [noise=true , address=45.91.116.10:4087 , piece_id=DBID6UJRTQPMNCQVZIKUUKHOJMQPICPRN6NUREDI3LW4HKJ6HMKA, seg_pos={0 0}, upload=133203]: Done uploading piece. err: <nil>
Dec 10 23:33:28.084288595 (128h5m28.911996976s) [noise=true , address=145.79.227.39:4021 , piece_id=PNWLJJV5PXQLBD4HV6SW4UGQF4VELIAFAIQMRR3NGQIJG3M54FTA, seg_pos={0 0}, upload=133203]: Done uploading piece. err: <nil>
Dec 10 23:33:28.085587813 (128h5m28.913296208s) [noise=true , address=145.79.227.3:4163 , piece_id=JWX5TES3CBKKV676VS2Q44AAY7RYJLWHMLMHPGOD3CPHSUW3CZCA, seg_pos={0 0}, upload=133203]: Done uploading piece. err: <nil>
Dec 10 23:33:28.086331807 (128h5m28.914040202s) [noise=true , address=145.79.227.22:4515 , piece_id=LXMGXZAGC6JNM7P7C67ZIDMJ2TRH52YT7OSKIHVN3ROU46L6CZPQ, seg_pos={0 0}, upload=133203]: Done uploading piece. err: <nil>
Dec 10 23:33:28.088632082 (128h5m28.916340477s) [noise=true , address=145.79.227.25:4647 , piece_id=BQZLQ5UEV6VOU6UVU3FZJ3FKPOVAOZQKD2EOK5O63EH4FEDNIEEA, seg_pos={0 0}, upload=133203]: Done uploading piece. err: <nil>
Dec 10 23:33:28.099190896 (128h5m28.926899291s) [noise=true , address=45.91.116.8:4072 , piece_id=WC6W6J6GKYYOGHAKIYQVWYAMBIFR2NR5PO4VM2ZNRDH535BPGVUA, seg_pos={0 0}, upload=133203]: Done uploading piece. err: <nil>
Dec 10 23:33:28.107355622 (128h5m28.935064014s) [noise=true , address=45.91.116.11:4138 , piece_id=V7SNI6FAPS4ZKH7KI32I7PJZWQHWJVUC5B6KFJGEZ7FV7IRXQZGQ, seg_pos={0 0}, upload=133203]: Done uploading piece. err: <nil>
Dec 10 23:33:28.121897580 (128h5m28.949605974s) [noise=true , address=145.79.227.38:4004 , piece_id=VGBBN4MUBMAZANLXKGEUD6HHPWNEXFXA2TNJBCGUCEWJ5BXTY5IQ, seg_pos={0 0}, upload=133203]: Done uploading piece. err: <nil>
Dec 10 23:33:28.182660039 (128h5m29.010368432s) [noise=true , address=5.231.73.4:4010 , piece_id=CTNYKYALWPEA7OMXZ5GSNAWW2C7WYL3GVQKERPZ6INPV44Y2LD7A, seg_pos={0 0}, upload=133203]: Done uploading piece. err: <nil>
Dec 10 23:33:28.217945328 (128h5m29.045653719s) [noise=true , address=45.91.116.11:4125 , piece_id=EE6IHZLI7QBKAF7KB4Y6BBQLJI5HQU42HRGMWN4PQMXGJ2G5U5NQ, seg_pos={0 0}, upload=133203]: Done uploading piece. err: <nil>
Dec 10 23:33:34.551890824 (128h5m35.379599223s) [noise=true , address=64.49.13.169:4709 , piece_id=B3A2ROTXNODVCTJMAMO5UGNBFJCTU4UBHD6NIHH6IVPGCM2VPSAA, seg_pos={0 0}, upload=133203]: Done uploading piece. err: <nil>
Dec 10 23:34:27.937958958 (128h6m28.765667414s) [noise=true , address=77.90.31.16:4764 , piece_id=EXZO4JXQL6WFI2ENZE2ZIKR5U2VQD7GGOQQM57ETGPI5VIO5ZLQQ, seg_pos={0 0}, upload=133203]: Done uploading piece. err: ecclient: upload failed (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT, address:77.90.31.16:4764): protocol: rpc: dial tcp 77.90.31.16:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:34:27.938135940 (128h6m28.765844339s) [noise=true , address=77.90.31.14:4673 , piece_id=AIVDB3XODE4FPD6SNWLGCKEXRI5JW62XKVMFL3QZDLT7GD4QWEVA, seg_pos={0 0}, upload=133203]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: rpc: dial tcp 77.90.31.14:4673: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:34:27.938288876 (128h6m28.765997273s) [noise=true , address=77.90.31.13:4635 , piece_id=D762FEUXE7HCL26VSIUV2VRNDGVDBY7SKQPC6GLHHL4YBZWV6H5A, seg_pos={0 0}, upload=133203]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:34:27.938317899 (128h6m28.766026302s) [noise=true , address=77.90.31.15:4707 , piece_id=7RAN7KV5KJLV2RGO4SOHANSJB32IYHEAWLY5737DDFTDY6KCTYUA, seg_pos={0 0}, upload=133203]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:34:27.938395379 (128h6m28.766103773s) [noise=true , address=77.90.31.19:4870 , piece_id=K4WCAKT6KYIL3AMNJSUG7IAROEDE5AP75PV3BBBWKV5DQ2IIEFQQ, seg_pos={0 0}, upload=133203]: Done uploading piece. err: ecclient: upload failed (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR, address:77.90.31.19:4870): protocol: write tcp 159.223.143.39:38720->77.90.31.19:4870: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:34:27.938516406 (128h6m28.766224807s) [noise=true , address=77.90.31.12:4608 , piece_id=JHJYWI3G53BYKDQMTL4T5EBN6NIXI5HI4OMXZNAQA5OXA5DEAGHQ, seg_pos={0 0}, upload=133203]: Done uploading piece. err: ecclient: upload failed (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, address:77.90.31.12:4608): protocol: rpc: dial tcp 77.90.31.12:4608: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:34:27.938536234 (128h6m28.766244625s) [seg_pos={0 0}, upload=133203]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:34:27.963521062 (128h6m28.791229453s) [upload=133203]: Done uploading segments...
Dec 10 23:34:27.963555840 (128h6m28.791264237s) [upload=133203]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1m0.027014609s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.............................................XXX............] duration:
596.348626ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:34:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 596.450588ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
10 out of 60 is failed (threshold=3) [.................................................XXXXXXX.XXX] duration:
1m0.026842209s trace:
https://jaeger.dev.storj.io/trace/293f96b3c90b89b0
FAIL (2025-12-10 23:34:21 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
m_ObjectBegin
Dec 10 23:33:21.593707886 (128h5m22.421416283s) [upload=133198]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:33:21.628593191 (128h5m22.456301571s) [seg_pos={0 0}, upload=133198]: Begin upload segment...
Dec 10 23:33:21.628652578 (128h5m22.456360969s) [seg_pos={0 0}, upload=133198]: Done begin upload segment.
Dec 10 23:33:21.628697456 (128h5m22.456405848s) [noise=true , address=45.91.116.8:4073 , piece_id=2XYISLWCCP44BLCZQNPSZ6VFT6N26KDAOKZURF4ILHKLV2QYKE5Q, seg_pos={0 0}, upload=133198]: Uploading piece...
Dec 10 23:33:21.628719883 (128h5m22.456428277s) [noise=true , address=77.90.31.18:4821 , piece_id=JN7WHGLBZ7ZGOGXAM3LSDNOVRTXTXTTS2VYRXTDCJIIF2KO3W5RQ, seg_pos={0 0}, upload=133198]: Uploading piece...
Dec 10 23:33:21.629163573 (128h5m22.456871971s) [noise=true , address=64.49.13.154:4930 , piece_id=TEWUZI5HXPXRZZV2WVTPJWBRFXUOPM6NRGUH3AKAWHZE46AJQ5ZQ, seg_pos={0 0}, upload=133198]: Uploading piece...
Dec 10 23:33:21.629513449 (128h5m22.457221843s) [noise=true , address=77.90.31.13:4630 , piece_id=3JYKMDLQRCUNYZP2TIMDJM7OJRUDJYXTAKSDLOX7HW2POXRJVA6Q, seg_pos={0 0}, upload=133198]: Uploading piece...
Dec 10 23:33:21.629620589 (128h5m22.457328986s) [noise=true , address=77.90.31.10:4524 , piece_id=V3IXSEEYIMF7L5NISRGFQ3SJLEENADYUID3GPSNPDWLHURWJLTVA, seg_pos={0 0}, upload=133198]: Uploading piece...
Dec 10 23:33:21.629880096 (128h5m22.457588491s) [noise=true , address=5.231.73.2:4019 , piece_id=RIDYN35LFBZJYTO2MAFVEDDFDBWVOHEYNWNL5CJQ2VYK3JUQRREQ, seg_pos={0 0}, upload=133198]: Uploading piece...
Dec 10 23:33:21.630081546 (128h5m22.457789968s) [noise=true , address=64.49.13.169:4709 , piece_id=FDIMYHXE5SQ5PVEXP5UGVYU3DZL3AB4XIX5AW7XSIQYHEZVZBVZA, seg_pos={0 0}, upload=133198]: Uploading piece...
Dec 10 23:33:21.630164103 (128h5m22.457872497s) [noise=true , address=145.79.227.30:4766 , piece_id=236OQJTQXWVAHPUH4TIJKQ5E3MKZY7HF2DBEOHWVLSCGJSUAOANA, seg_pos={0 0}, upload=133198]: Uploading piece...
Dec 10 23:33:21.630473557 (128h5m22.458181952s) [noise=true , address=145.79.227.34:4910 , piece_id=SFURIZWSDFKZP7O3GPDHFJB7UK6RRHOA2IPHIRIZGDLRWQLC4T2A, seg_pos={0 0}, upload=133198]: Uploading piece...
Dec 10 23:33:21.630554719 (128h5m22.4582631s) [noise=true , address=5.231.73.5:4035 , piece_id=TGQ33JE5BP5JAEPUT4JO5TK3VYJIYDZKLSS43W2CFRHRRF6OAINQ, seg_pos={0 0}, upload=133198]: Uploading piece...
Dec 10 23:33:21.630845046 (128h5m22.458553447s) [noise=true , address=145.79.227.3:4183 , piece_id=AZ34XMWWDONJ77X3Q6W3NFFBMLNIL5RHBGXO3V3RURW2EZDZS5OA, seg_pos={0 0}, upload=133198]: Uploading piece...
Dec 10 23:33:21.630967671 (128h5m22.458676063s) [noise=true , address=45.91.116.7:4044 , piece_id=XOFNEYYWAWBMJXPQDQQUL2OT4GFNG6OT2AANS7QVDZQ66WPJL42Q, seg_pos={0 0}, upload=133198]: Uploading piece...
Dec 10 23:33:21.631330131 (128h5m22.459038526s) [noise=true , address=145.79.227.35:4510 , piece_id=H5FA7KLT6SEWJNA2V6MXO4HMUEVMRSFC3TDYXLM4COFE7AFZC7RA, seg_pos={0 0}, upload=133198]: Uploading piece...
Dec 10 23:33:21.631411285 (128h5m22.459119685s) [noise=true , address=145.79.227.16:4919 , piece_id=TMGBILJVN6HBHORHNOI2A67DNMMEBL76SN2NKBIR5DG6ARCS52BQ, seg_pos={0 0}, upload=133198]: Uploading piece...
Dec 10 23:33:21.631674582 (128h5m22.45938298s) [noise=true , address=77.90.31.16:4752 , piece_id=ARN3OMERWF7NCK2VUZLIVFZFB4VZ6LUNUS7FAACQPR352QMTJP6Q, seg_pos={0 0}, upload=133198]: Uploading piece...
Dec 10 23:33:21.631850958 (128h5m22.459559352s) [noise=true , address=77.90.31.19:4850 , piece_id=PSBN376T2D2R6U42NSQLHAQLUIFBQK6O2FH3WCNWQRI57GUGUTTQ, seg_pos={0 0}, upload=133198]: Uploading piece...
Dec 10 23:33:21.632289383 (128h5m22.459997774s) [noise=true , address=77.90.31.11:4575 , piece_id=ON2UXNPOOB6AUIODJ4375OH3YFKSRZDAWAEGRRUF57E52IC2AHIQ, seg_pos={0 0}, upload=133198]: Uploading piece...
Dec 10 23:33:21.632805085 (128h5m22.460513482s) [noise=true , address=145.79.227.31:4824 , piece_id=HY2A72KRXVO5UMOWAMLKVNAS3ZBEZIBZXXQ24Z5LYXMQUQONYCNA, seg_pos={0 0}, upload=133198]: Uploading piece...
Dec 10 23:33:21.696651041 (128h5m22.524359442s) [upload=133198]: Next returned nil segment
Dec 10 23:33:21.696666167 (128h5m22.524374561s) [upload=133198]: Waiting for error group managing segments...
Dec 10 23:33:21.703850641 (128h5m22.531559036s) [noise=true , address=145.79.227.35:4510 , piece_id=H5FA7KLT6SEWJNA2V6MXO4HMUEVMRSFC3TDYXLM4COFE7AFZC7RA, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil>
Dec 10 23:33:21.711668142 (128h5m22.539376524s) [noise=true , address=145.79.227.34:4910 , piece_id=SFURIZWSDFKZP7O3GPDHFJB7UK6RRHOA2IPHIRIZGDLRWQLC4T2A, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil>
Dec 10 23:33:21.717634735 (128h5m22.545343129s) [noise=true , address=45.91.116.8:4073 , piece_id=2XYISLWCCP44BLCZQNPSZ6VFT6N26KDAOKZURF4ILHKLV2QYKE5Q, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil>
Dec 10 23:33:21.718530950 (128h5m22.546239341s) [noise=true , address=145.79.227.3:4183 , piece_id=AZ34XMWWDONJ77X3Q6W3NFFBMLNIL5RHBGXO3V3RURW2EZDZS5OA, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil>
Dec 10 23:33:21.722796012 (128h5m22.550504411s) [noise=true , address=45.91.116.7:4044 , piece_id=XOFNEYYWAWBMJXPQDQQUL2OT4GFNG6OT2AANS7QVDZQ66WPJL42Q, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil>
Dec 10 23:33:21.746909276 (128h5m22.574617657s) [noise=true , address=145.79.227.31:4824 , piece_id=HY2A72KRXVO5UMOWAMLKVNAS3ZBEZIBZXXQ24Z5LYXMQUQONYCNA, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil>
Dec 10 23:33:21.781100457 (128h5m22.608808849s) [noise=true , address=5.231.73.2:4019 , piece_id=RIDYN35LFBZJYTO2MAFVEDDFDBWVOHEYNWNL5CJQ2VYK3JUQRREQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil>
Dec 10 23:33:21.795649473 (128h5m22.623357864s) [noise=true , address=5.231.73.5:4035 , piece_id=TGQ33JE5BP5JAEPUT4JO5TK3VYJIYDZKLSS43W2CFRHRRF6OAINQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil>
Dec 10 23:33:22.244265861 (128h5m23.071974253s) [noise=true , address=64.49.13.154:4930 , piece_id=TEWUZI5HXPXRZZV2WVTPJWBRFXUOPM6NRGUH3AKAWHZE46AJQ5ZQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil>
Dec 10 23:33:22.678957582 (128h5m23.506665976s) [noise=true , address=145.79.227.30:4766 , piece_id=236OQJTQXWVAHPUH4TIJKQ5E3MKZY7HF2DBEOHWVLSCGJSUAOANA, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil>
Dec 10 23:33:22.743509004 (128h5m23.571217401s) [noise=true , address=145.79.227.16:4919 , piece_id=TMGBILJVN6HBHORHNOI2A67DNMMEBL76SN2NKBIR5DG6ARCS52BQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil>
Dec 10 23:33:29.093123715 (128h5m29.920832096s) [noise=true , address=64.49.13.169:4709 , piece_id=FDIMYHXE5SQ5PVEXP5UGVYU3DZL3AB4XIX5AW7XSIQYHEZVZBVZA, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil>
Dec 10 23:34:21.562878797 (128h6m22.390587195s) [noise=true , address=77.90.31.19:4850 , piece_id=PSBN376T2D2R6U42NSQLHAQLUIFBQK6O2FH3WCNWQRI57GUGUTTQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: write tcp 159.223.143.39:49186->77.90.31.19:4850: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:34:21.563054641 (128h6m22.390763041s) [noise=true , address=77.90.31.18:4821 , piece_id=JN7WHGLBZ7ZGOGXAM3LSDNOVRTXTXTTS2VYRXTDCJIIF2KO3W5RQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:34:21.563238700 (128h6m22.3909471s) [noise=true , address=77.90.31.13:4630 , piece_id=3JYKMDLQRCUNYZP2TIMDJM7OJRUDJYXTAKSDLOX7HW2POXRJVA6Q, seg_pos={0 0}, upload=133198]: Done uploading piece. err: ecclient: upload failed (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn, address:77.90.31.13:4630): protocol: rpc: dial tcp 77.90.31.13:4630: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:34:21.563250407 (128h6m22.390958805s) [noise=true , address=77.90.31.16:4752 , piece_id=ARN3OMERWF7NCK2VUZLIVFZFB4VZ6LUNUS7FAACQPR352QMTJP6Q, seg_pos={0 0}, upload=133198]: Done uploading piece. err: ecclient: upload failed (node:12iLnZ6bvgFWKtP79m7pa6rmStrpMGJQeUuDV3kTHQdFZQAWS4, address:77.90.31.16:4752): protocol: rpc: dial tcp 77.90.31.16:4752: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:34:21.563390697 (128h6m22.391099095s) [noise=true , address=77.90.31.11:4575 , piece_id=ON2UXNPOOB6AUIODJ4375OH3YFKSRZDAWAEGRRUF57E52IC2AHIQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 159.223.143.39:50808->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:34:21.563448994 (128h6m22.391157371s) [noise=true , address=77.90.31.10:4524 , piece_id=V3IXSEEYIMF7L5NISRGFQ3SJLEENADYUID3GPSNPDWLHURWJLTVA, seg_pos={0 0}, upload=133198]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: rpc: dial tcp 77.90.31.10:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:34:21.563469675 (128h6m22.391178066s) [seg_pos={0 0}, upload=133198]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:34:21.587649605 (128h6m22.415358007s) [upload=133198]: Done uploading segments...
Dec 10 23:34:21.587699293 (128h6m22.415407687s) [upload=133198]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1m0.026859305s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [............................................X....X...X......] duration:
455.764752ms trace:
https://jaeger.dev.storj.io/trace/749247032308b962
FAIL (2025-12-10 23:34:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 455.790779ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [............................................XXX.X...........] duration:
485.975299ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:34:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 486.009108ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..............................................XXX...........] duration:
569.741978ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:33:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 569.784194ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...............................................XXX..........] duration:
585.587031ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:33:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 585.612362ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [..................................................XXXXXXX.XX] duration:
1m0.022891553s trace:
https://jaeger.dev.storj.io/trace/66e3c7683bc5c520
FAIL (2025-12-10 23:33:21 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
0}, upload=133181]: Done begin upload segment.
Dec 10 23:32:21.531659821 (128h4m22.359368216s) [noise=true , address=45.91.116.7:4027 , piece_id=TPT7TDHTLTCRA3DII6FIXZFSNHRQ62WMOTX53UWZTLXLD4RR2JUA, seg_pos={0 0}, upload=133181]: Uploading piece...
Dec 10 23:32:21.531703440 (128h4m22.35941184s) [noise=true , address=77.90.31.16:4738 , piece_id=36ASCVQOUTUQJDFWTWOTR2BPOGW3G6XD27T6N2VGSAZPUATOYLCA, seg_pos={0 0}, upload=133181]: Uploading piece...
Dec 10 23:32:21.532289907 (128h4m22.359998308s) [noise=true , address=77.90.31.16:4767 , piece_id=HCN6CNASBUDSAOPRSKVVXGNRX5HU56SL76KMXLVA4F7MWCNKWWUA, seg_pos={0 0}, upload=133181]: Uploading piece...
Dec 10 23:32:21.532730754 (128h4m22.360439151s) [noise=true , address=45.91.116.9:4293 , piece_id=Z55CBK37HQR2HVIMVOCJ5ZUM5XFBKPZCRPF52OA4AKQSMAN2IZ4Q, seg_pos={0 0}, upload=133181]: Uploading piece...
Dec 10 23:32:21.533049474 (128h4m22.360757871s) [noise=true , address=77.90.31.18:4818 , piece_id=3K7PKT4YBEKTES2GLMLLEFUI6B577FASG2FPH3W47HP6HLPB5Y7Q, seg_pos={0 0}, upload=133181]: Uploading piece...
Dec 10 23:32:21.533155699 (128h4m22.36086409s) [noise=true , address=5.231.73.4:4011 , piece_id=NK3FIYN3OB3ICF7G75S2XQGJ66IDWD473HI3QTE4Z55HZEKDWGGA, seg_pos={0 0}, upload=133181]: Uploading piece...
Dec 10 23:32:21.533552593 (128h4m22.361260987s) [noise=true , address=64.49.13.170:4722 , piece_id=JM252WIK3RPUXSA3PQ7YZDJ6C7J4BHLK2RJOSCMEC4K27CT2FKYA, seg_pos={0 0}, upload=133181]: Uploading piece...
Dec 10 23:32:21.533587796 (128h4m22.36129619s) [noise=true , address=45.91.116.5:4310 , piece_id=TL2N34GCSDF5DMJFTQNCQZUVTPI5N23N6FL6ZIWX34ZKX3PRYCSA, seg_pos={0 0}, upload=133181]: Uploading piece...
Dec 10 23:32:21.533906425 (128h4m22.361614819s) [noise=true , address=145.79.227.20:4471 , piece_id=3CWWHGY5D7LQ27VPSWAPZ5JDO7UQ3DQ4M62B7T6F7LZ2IGLJ34YQ, seg_pos={0 0}, upload=133181]: Uploading piece...
Dec 10 23:32:21.534104027 (128h4m22.361812429s) [noise=true , address=77.90.31.15:4714 , piece_id=X3SM24S7FC2XTSWTQ57E5SHVUARPF555HBU5PYXS5JBAH6FUD4EQ, seg_pos={0 0}, upload=133181]: Uploading piece...
Dec 10 23:32:21.534251291 (128h4m22.361959714s) [noise=true , address=45.91.116.4:4942 , piece_id=6FHVO3EZ4A4H777TJI3DSPCEDHF2TH2CM3523WK5MCI5JIP5WEBQ, seg_pos={0 0}, upload=133181]: Uploading piece...
Dec 10 23:32:21.534622677 (128h4m22.362331071s) [noise=true , address=145.79.227.41:4113 , piece_id=LQ5JXZEHHBLSPQXJZIJNNAFDT2JFSJOUS3RZCJJ7RB7C2BH25DLQ, seg_pos={0 0}, upload=133181]: Uploading piece...
Dec 10 23:32:21.534654470 (128h4m22.362362861s) [noise=true , address=77.90.31.14:4687 , piece_id=CX7DR5BWW4IYP7LTKJN2N7GOCSC77UOGW3EQN6LTKMGU3PAVGROA, seg_pos={0 0}, upload=133181]: Uploading piece...
Dec 10 23:32:21.535149053 (128h4m22.362857444s) [noise=true , address=145.79.227.16:4919 , piece_id=JPVS6AE2AOVV63RLK7VVODLJEJQBT4SSZEBHAWUAPKHBIQLJK7QA, seg_pos={0 0}, upload=133181]: Uploading piece...
Dec 10 23:32:21.535316914 (128h4m22.363025306s) [noise=true , address=145.79.227.30:4779 , piece_id=OKZVJU7RSB4CEBBOJC6VNYH2TCEGJVHXD4F2CAX6YGLEIGKRU2WA, seg_pos={0 0}, upload=133181]: Uploading piece...
Dec 10 23:32:21.535645631 (128h4m22.363354023s) [noise=true , address=77.90.31.10:4548 , piece_id=NMRJFKGLH7EGGPGYILH5VWWO4TT6KWTA7GRAECSQG4FCC52CC2FA, seg_pos={0 0}, upload=133181]: Uploading piece...
Dec 10 23:32:21.535842914 (128h4m22.363551308s) [noise=true , address=77.90.31.13:4639 , piece_id=5TLGJWS3IUAULTJAPPRA7E4IH5OEXAXPRZVOCG7EGF22UIUKE2ZQ, seg_pos={0 0}, upload=133181]: Uploading piece...
Dec 10 23:32:21.536620270 (128h4m22.364328661s) [noise=true , address=77.90.31.11:4564 , piece_id=TT7UIIHQU3YQ3JEQC5XUGQO2OI5IC72DUCBCMVYZO6DNGV2ESAOQ, seg_pos={0 0}, upload=133181]: Uploading piece...
Dec 10 23:32:21.598423877 (128h4m22.426132259s) [upload=133181]: Next returned nil segment
Dec 10 23:32:21.598438543 (128h4m22.426146934s) [upload=133181]: Waiting for error group managing segments...
Dec 10 23:32:21.603042842 (128h4m22.430751246s) [noise=true , address=45.91.116.9:4293 , piece_id=Z55CBK37HQR2HVIMVOCJ5ZUM5XFBKPZCRPF52OA4AKQSMAN2IZ4Q, seg_pos={0 0}, upload=133181]: Done uploading piece. err: <nil>
Dec 10 23:32:21.603255605 (128h4m22.430963996s) [noise=true , address=45.91.116.4:4942 , piece_id=6FHVO3EZ4A4H777TJI3DSPCEDHF2TH2CM3523WK5MCI5JIP5WEBQ, seg_pos={0 0}, upload=133181]: Done uploading piece. err: <nil>
Dec 10 23:32:21.604664248 (128h4m22.432372639s) [noise=true , address=145.79.227.41:4113 , piece_id=LQ5JXZEHHBLSPQXJZIJNNAFDT2JFSJOUS3RZCJJ7RB7C2BH25DLQ, seg_pos={0 0}, upload=133181]: Done uploading piece. err: <nil>
Dec 10 23:32:21.618182956 (128h4m22.445891337s) [noise=true , address=45.91.116.5:4310 , piece_id=TL2N34GCSDF5DMJFTQNCQZUVTPI5N23N6FL6ZIWX34ZKX3PRYCSA, seg_pos={0 0}, upload=133181]: Done uploading piece. err: <nil>
Dec 10 23:32:21.618966822 (128h4m22.446675213s) [noise=true , address=145.79.227.20:4471 , piece_id=3CWWHGY5D7LQ27VPSWAPZ5JDO7UQ3DQ4M62B7T6F7LZ2IGLJ34YQ, seg_pos={0 0}, upload=133181]: Done uploading piece. err: <nil>
Dec 10 23:32:21.750664717 (128h4m22.578373108s) [noise=true , address=45.91.116.7:4027 , piece_id=TPT7TDHTLTCRA3DII6FIXZFSNHRQ62WMOTX53UWZTLXLD4RR2JUA, seg_pos={0 0}, upload=133181]: Done uploading piece. err: <nil>
Dec 10 23:32:21.795322579 (128h4m22.623030975s) [noise=true , address=5.231.73.4:4011 , piece_id=NK3FIYN3OB3ICF7G75S2XQGJ66IDWD473HI3QTE4Z55HZEKDWGGA, seg_pos={0 0}, upload=133181]: Done uploading piece. err: <nil>
Dec 10 23:32:22.123306061 (128h4m22.951014456s) [noise=true , address=64.49.13.170:4722 , piece_id=JM252WIK3RPUXSA3PQ7YZDJ6C7J4BHLK2RJOSCMEC4K27CT2FKYA, seg_pos={0 0}, upload=133181]: Done uploading piece. err: <nil>
Dec 10 23:32:22.623227384 (128h4m23.450935776s) [noise=true , address=145.79.227.30:4779 , piece_id=OKZVJU7RSB4CEBBOJC6VNYH2TCEGJVHXD4F2CAX6YGLEIGKRU2WA, seg_pos={0 0}, upload=133181]: Done uploading piece. err: <nil>
Dec 10 23:32:22.672011278 (128h4m23.49971967s) [noise=true , address=145.79.227.16:4919 , piece_id=JPVS6AE2AOVV63RLK7VVODLJEJQBT4SSZEBHAWUAPKHBIQLJK7QA, seg_pos={0 0}, upload=133181]: Done uploading piece. err: <nil>
Dec 10 23:33:21.459835610 (128h5m22.287544011s) [noise=true , address=77.90.31.14:4687 , piece_id=CX7DR5BWW4IYP7LTKJN2N7GOCSC77UOGW3EQN6LTKMGU3PAVGROA, seg_pos={0 0}, upload=133181]: Done uploading piece. err: ecclient: upload failed (node:1UydYeC9AEPHMXQZGaDhgS2vNguG4umL5nnBLpty2L3zeQAvLu, address:77.90.31.14:4687): protocol: rpc: dial tcp 77.90.31.14:4687: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:33:21.459934235 (128h5m22.287642636s) [noise=true , address=77.90.31.16:4767 , piece_id=HCN6CNASBUDSAOPRSKVVXGNRX5HU56SL76KMXLVA4F7MWCNKWWUA, seg_pos={0 0}, upload=133181]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:33:21.460174115 (128h5m22.287882513s) [noise=true , address=77.90.31.15:4714 , piece_id=X3SM24S7FC2XTSWTQ57E5SHVUARPF555HBU5PYXS5JBAH6FUD4EQ, seg_pos={0 0}, upload=133181]: Done uploading piece. err: ecclient: upload failed (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP, address:77.90.31.15:4714): protocol: rpc: dial tcp 77.90.31.15:4714: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:33:21.460299170 (128h5m22.288007572s) [noise=true , address=77.90.31.13:4639 , piece_id=5TLGJWS3IUAULTJAPPRA7E4IH5OEXAXPRZVOCG7EGF22UIUKE2ZQ, seg_pos={0 0}, upload=133181]: Done uploading piece. err: ecclient: upload failed (node:12N9N1pcaVn3nPh1WY7uktz8Z7fwJGqnTdVwZMAW5SVrqvRA8VV, address:77.90.31.13:4639): protocol: rpc: dial tcp 77.90.31.13:4639: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:33:21.460431868 (128h5m22.288140268s) [noise=true , address=77.90.31.10:4548 , piece_id=NMRJFKGLH7EGGPGYILH5VWWO4TT6KWTA7GRAECSQG4FCC52CC2FA, seg_pos={0 0}, upload=133181]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:33:21.460708051 (128h5m22.288416469s) [noise=true , address=77.90.31.18:4818 , piece_id=3K7PKT4YBEKTES2GLMLLEFUI6B577FASG2FPH3W47HP6HLPB5Y7Q, seg_pos={0 0}, upload=133181]: Done uploading piece. err: ecclient: upload failed (node:12LuFBTJg1aNmZoEdiQTtvfnfh5qnWVZFtqFX93L4fNfiYTYqJ6, address:77.90.31.18:4818): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:33:21.460972735 (128h5m22.288681126s) [noise=true , address=77.90.31.16:4738 , piece_id=36ASCVQOUTUQJDFWTWOTR2BPOGW3G6XD27T6N2VGSAZPUATOYLCA, seg_pos={0 0}, upload=133181]: Done uploading piece. err: ecclient: upload failed (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd, address:77.90.31.16:4738): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:33:21.461187853 (128h5m22.288896247s) [noise=true , address=77.90.31.11:4564 , piece_id=TT7UIIHQU3YQ3JEQC5XUGQO2OI5IC72DUCBCMVYZO6DNGV2ESAOQ, seg_pos={0 0}, upload=133181]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:55626->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:33:21.461218556 (128h5m22.28892696s) [seg_pos={0 0}, upload=133181]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:33:21.480915170 (128h5m22.308623561s) [upload=133181]: Done uploading segments...
Dec 10 23:33:21.481090226 (128h5m22.308798617s) [upload=133181]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 1m0.022938284s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.............................................XXX.X..........] duration:
418.648177ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:33:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 418.680925ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.............................................X....X...X.....] duration:
528.694253ms trace:
https://jaeger.dev.storj.io/trace/749247032308b962
FAIL (2025-12-10 23:33:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 528.761686ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..............................................X....X...X....] duration:
463.333638ms trace:
https://jaeger.dev.storj.io/trace/749247032308b962
FAIL (2025-12-10 23:32:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 463.390513ms
edge
gateway
us-select-1
|