Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.................................................XX.........] duration:
563.807311ms
FAIL (2025-07-11 22:14:35 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 563.833876ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
817.117218ms
FAIL (2025-07-11 22:14:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 817.171083ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [............................................XX..XXXX........] duration:
650.306454ms
FAIL (2025-07-11 22:14:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 650.331383ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.......................................X..XX.XXXXX..........] duration:
539.80167ms
FAIL (2025-07-11 22:13:59 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 539.831086ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..................................................XX........] duration:
752.643789ms
FAIL (2025-07-11 22:13:45 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 752.675485ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [........................................X..XX.XXXXX.........] duration:
484.486818ms
FAIL (2025-07-11 22:13:43 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 484.516741ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
676.927309ms
FAIL (2025-07-11 22:13:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 676.976776ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...................................................XX.......] duration:
814.397753ms
FAIL (2025-07-11 22:13:21 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 814.430491ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.............................................XX..XXXX.......] duration:
510.215975ms
FAIL (2025-07-11 22:13:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 510.240234ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.........................................X..XX.XXXXX........] duration:
445.807878ms
FAIL (2025-07-11 22:12:50 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 445.837849ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..............................................XX..XXXX......] duration:
615.345067ms
FAIL (2025-07-11 22:12:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 615.387241ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
702.245521ms
FAIL (2025-07-11 22:12:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 702.278893ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [..........................................X..XX.XXXXX.......] duration:
472.197659ms
FAIL (2025-07-11 22:12:17 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 472.230776ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [....................................................XX......] duration:
697.034157ms
FAIL (2025-07-11 22:12:11 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 697.085995ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [...............................................XX..XXXX.....] duration:
680.126526ms
FAIL (2025-07-11 22:11:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 680.153275ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [...........................................X..XX.XXXXX......] duration:
621.50165ms
FAIL (2025-07-11 22:11:41 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 621.535959ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
753.625588ms
FAIL (2025-07-11 22:11:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 753.711613ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [............................................X..XX.XXXXX.....] duration:
744.949243ms
FAIL (2025-07-11 22:11:24 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 744.991852ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
4.465250185s
FAIL (2025-07-11 22:11:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.465301229s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration:
1.260167745s
FAIL (2025-07-11 22:11:11 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1.260230196s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [................................................XX..XXXX....] duration:
683.457967ms
FAIL (2025-07-11 22:11:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 683.517483ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [......................................................XX....] duration:
936.251857ms
FAIL (2025-07-11 22:10:47 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 936.304214ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.............................................X..XX.XXXXX....] duration:
538.212051ms
FAIL (2025-07-11 22:10:40 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 538.244302ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
2.380174633s
FAIL (2025-07-11 22:10:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.380198354s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [..............................................X..XX.XXXXX...] duration:
1.499804193s
FAIL (2025-07-11 22:09:53 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 1.499885693s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.......................................................XX...] duration:
1.471737295s
FAIL (2025-07-11 22:09:52 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1.47176488s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.................................................XX..XXXX...] duration:
2.86797163s
FAIL (2025-07-11 22:09:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.868793231s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
6.249966434s
FAIL (2025-07-11 22:09:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.250014812s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [...............................................X..XX.XXXXX..] duration:
8.595960535s
FAIL (2025-07-11 22:09:30 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 8.596008032s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..................................................XX..XXXX..] duration:
3.741129485s
FAIL (2025-07-11 22:09:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.741206585s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [................................................X..XX.XXXXX.] duration:
25.424540686s
FAIL (2025-07-11 22:09:21 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 25.424572964s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [...................................................XX..XXXX.] duration:
12.185156328s
FAIL (2025-07-11 22:09:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 12.185187512s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [X...................................................XX..XXXX] duration:
30.00071558s
FAIL (2025-07-11 22:08:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000741802s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [........................................................XX..] duration:
630.553888ms
FAIL (2025-07-11 22:08:47 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 630.591954ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
619.735069ms
FAIL (2025-07-11 22:08:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 619.786613ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.................................................X..XX.XXXXX] duration:
30.031304231s
FAIL (2025-07-11 22:08:19 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 30.031330678s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.........................................................XX.] duration:
1.910450536s
FAIL (2025-07-11 22:07:50 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1.910496276s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
1m0.028506438s
FAIL (2025-07-11 22:07:48 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1m0.028522883s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [XX...................................................XX..XXX] duration:
30.000745453s
FAIL (2025-07-11 22:07:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000788407s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
2.704879734s
FAIL (2025-07-11 22:07:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.704910187s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [..................................................X..XX.XXXX] duration:
30.02907087s
FAIL (2025-07-11 22:07:23 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
) [upload=356247]: Got next segment. Inline: false
Jul 11 22:06:53.213038576 (444h55m20.139333265s) [upload=356247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:06:53.213046522 (444h55m20.139341209s) [upload=356247]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:53.241916770 (444h55m20.16821146s) [seg_pos={0 0}, upload=356247]: Begin upload segment...
Jul 11 22:06:53.241981852 (444h55m20.168276545s) [seg_pos={0 0}, upload=356247]: Done begin upload segment.
Jul 11 22:06:53.242014670 (444h55m20.168309368s) [noise=true , address=104.36.82.11:4114 , piece_id=HPWVKKK3RXNJNUBGYGBB2OBL76YK7G2IABBQTX4U75TUBG7OPCWQ, seg_pos={0 0}, upload=356247]: Uploading piece...
Jul 11 22:06:53.242063172 (444h55m20.168357867s) [noise=true , address=104.36.82.13:4186 , piece_id=W2IEKMHUS7E7L2HGF45SD422T4XBGGHTAUTMGTOBTN4FDQ5EVMIQ, seg_pos={0 0}, upload=356247]: Uploading piece...
Jul 11 22:06:53.242500440 (444h55m20.168795137s) [noise=true , address=104.36.82.14:4223 , piece_id=HBBHH4GTE4P3JY5KR5L42OPSBZV7Q7G7M4L564D2TDQPEV7MLLWQ, seg_pos={0 0}, upload=356247]: Uploading piece...
Jul 11 22:06:53.242542265 (444h55m20.168836955s) [noise=true , address=104.36.82.5:4837 , piece_id=E2ZRVYPEPOYUIDDXFYUBAQCKN2BSAKYR5NQXJUQHXUPF3GGUVQ2A, seg_pos={0 0}, upload=356247]: Uploading piece...
Jul 11 22:06:53.242938767 (444h55m20.169233457s) [noise=true , address=104.36.82.9:4007 , piece_id=43MZ3TZPPUQZQ2HBMZG3YHLHG5JCOSP3PA6PN6JKFMCRIB63VNUQ, seg_pos={0 0}, upload=356247]: Uploading piece...
Jul 11 22:06:53.243082536 (444h55m20.16937724s) [noise=true , address=77.90.31.14:4661 , piece_id=7MPTD63T2EEFUTLHRZVBOXCFOLLXO7O77ZXUQ4Z5IL4KGIWGFIMQ, seg_pos={0 0}, upload=356247]: Uploading piece...
Jul 11 22:06:53.243334075 (444h55m20.169628761s) [noise=true , address=77.90.31.11:4555 , piece_id=TDLSD63LNZR6GBWHPWRHOCOGVHALEXYQFNWBZPCPRE44TRRJRBEQ, seg_pos={0 0}, upload=356247]: Uploading piece...
Jul 11 22:06:53.243530070 (444h55m20.169824771s) [noise=true , address=104.36.82.4:4803 , piece_id=7MSIWKFUIKVUANRYUKBRFNLVS6L7ZVFYMT6WB4TWBRPZEMOW34SQ, seg_pos={0 0}, upload=356247]: Uploading piece...
Jul 11 22:06:53.243712390 (444h55m20.170007082s) [noise=true , address=104.36.82.6:4876 , piece_id=SGCZCUVJI45J2IKLODRMMJC24MC65RHF6LGZVHTQYSZEFQ62LKGA, seg_pos={0 0}, upload=356247]: Uploading piece...
Jul 11 22:06:53.243944298 (444h55m20.170239006s) [noise=true , address=104.36.82.12:4149 , piece_id=55734NNDMDA3CS5DPF7AVCIYXMS5RBA6HHJT5OCG5Q6LEBVPUZRA, seg_pos={0 0}, upload=356247]: Uploading piece...
Jul 11 22:06:53.244181842 (444h55m20.170476539s) [noise=true , address=45.91.116.4:4937 , piece_id=BBBCGBJQVCWDGPLXZQ3RYXNQ4KEAR2NJFT6WP7LOTTQUKVEVJTRA, seg_pos={0 0}, upload=356247]: Uploading piece...
Jul 11 22:06:53.244363860 (444h55m20.170658557s) [noise=true , address=45.91.116.14:4191 , piece_id=ZZ5SNEMHIWHWD6FZDELNNBA2VQSHBF3LH2RLF46G674QNYW5VXBA, seg_pos={0 0}, upload=356247]: Uploading piece...
Jul 11 22:06:53.244743761 (444h55m20.171038456s) [noise=true , address=104.36.82.15:4259 , piece_id=D26DTTLWDTLRFFLVA3JU7D5NDNBJQLPT4FEDCQNSLHRYO47GZNBQ, seg_pos={0 0}, upload=356247]: Uploading piece...
Jul 11 22:06:53.245641183 (444h55m20.171935872s) [noise=true , address=77.90.31.10:4517 , piece_id=SXI4YWXUZ5MTQZRGFDV7WZSM7K2LSII2F7ZRMARIJQ4OA4TVZMCQ, seg_pos={0 0}, upload=356247]: Uploading piece...
Jul 11 22:06:53.247160395 (444h55m20.173455083s) [noise=true , address=45.91.116.7:4011 , piece_id=5PMCST226HOPRU2LH4A47B3HJHQTPJUJ5ZCQ2LF5W7SZY5IJTHBQ, seg_pos={0 0}, upload=356247]: Uploading piece...
Jul 11 22:06:53.247540219 (444h55m20.173834907s) [noise=true , address=45.91.116.5:4299 , piece_id=26X7UKTBNTNZPZPUSF47ZFH2PZHDP7HEF4BLFZXGF3WXODFEZ5HQ, seg_pos={0 0}, upload=356247]: Uploading piece...
Jul 11 22:06:53.327069513 (444h55m20.253364189s) [upload=356247]: Next returned nil segment
Jul 11 22:06:53.327089283 (444h55m20.253383977s) [upload=356247]: Waiting for error group managing segments...
Jul 11 22:06:53.332335186 (444h55m20.258629866s) [noise=true , address=45.91.116.7:4011 , piece_id=5PMCST226HOPRU2LH4A47B3HJHQTPJUJ5ZCQ2LF5W7SZY5IJTHBQ, seg_pos={0 0}, upload=356247]: Done uploading piece. err: <nil>
Jul 11 22:06:53.332756447 (444h55m20.259051127s) [noise=true , address=45.91.116.4:4937 , piece_id=BBBCGBJQVCWDGPLXZQ3RYXNQ4KEAR2NJFT6WP7LOTTQUKVEVJTRA, seg_pos={0 0}, upload=356247]: Done uploading piece. err: <nil>
Jul 11 22:06:53.339109919 (444h55m20.265404612s) [noise=true , address=45.91.116.5:4299 , piece_id=26X7UKTBNTNZPZPUSF47ZFH2PZHDP7HEF4BLFZXGF3WXODFEZ5HQ, seg_pos={0 0}, upload=356247]: Done uploading piece. err: <nil>
Jul 11 22:06:53.343064572 (444h55m20.269359266s) [noise=true , address=45.91.116.14:4191 , piece_id=ZZ5SNEMHIWHWD6FZDELNNBA2VQSHBF3LH2RLF46G674QNYW5VXBA, seg_pos={0 0}, upload=356247]: Done uploading piece. err: <nil>
Jul 11 22:06:56.255721821 (444h55m23.182016508s) [noise=true , address=77.90.31.14:4661 , piece_id=7MPTD63T2EEFUTLHRZVBOXCFOLLXO7O77ZXUQ4Z5IL4KGIWGFIMQ, seg_pos={0 0}, upload=356247]: Done uploading piece. err: <nil>
Jul 11 22:06:58.442542035 (444h55m25.368836721s) [noise=true , address=77.90.31.11:4555 , piece_id=TDLSD63LNZR6GBWHPWRHOCOGVHALEXYQFNWBZPCPRE44TRRJRBEQ, seg_pos={0 0}, upload=356247]: Done uploading piece. err: <nil>
Jul 11 22:06:58.971425034 (444h55m25.897719706s) [noise=true , address=77.90.31.10:4517 , piece_id=SXI4YWXUZ5MTQZRGFDV7WZSM7K2LSII2F7ZRMARIJQ4OA4TVZMCQ, seg_pos={0 0}, upload=356247]: Done uploading piece. err: <nil>
Jul 11 22:07:23.184435141 (444h55m50.110729838s) [noise=true , address=104.36.82.5:4837 , piece_id=E2ZRVYPEPOYUIDDXFYUBAQCKN2BSAKYR5NQXJUQHXUPF3GGUVQ2A, seg_pos={0 0}, upload=356247]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:23.184627276 (444h55m50.110921967s) [noise=true , address=104.36.82.9:4007 , piece_id=43MZ3TZPPUQZQ2HBMZG3YHLHG5JCOSP3PA6PN6JKFMCRIB63VNUQ, seg_pos={0 0}, upload=356247]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:23.184695626 (444h55m50.110990316s) [noise=true , address=104.36.82.6:4876 , piece_id=SGCZCUVJI45J2IKLODRMMJC24MC65RHF6LGZVHTQYSZEFQ62LKGA, seg_pos={0 0}, upload=356247]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:23.184765159 (444h55m50.111059847s) [noise=true , address=104.36.82.12:4149 , piece_id=55734NNDMDA3CS5DPF7AVCIYXMS5RBA6HHJT5OCG5Q6LEBVPUZRA, seg_pos={0 0}, upload=356247]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:23.184840994 (444h55m50.111135683s) [noise=true , address=104.36.82.13:4186 , piece_id=W2IEKMHUS7E7L2HGF45SD422T4XBGGHTAUTMGTOBTN4FDQ5EVMIQ, seg_pos={0 0}, upload=356247]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 159.223.143.39:37142->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:23.185226908 (444h55m50.111521598s) [noise=true , address=104.36.82.11:4114 , piece_id=HPWVKKK3RXNJNUBGYGBB2OBL76YK7G2IABBQTX4U75TUBG7OPCWQ, seg_pos={0 0}, upload=356247]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:23.185304033 (444h55m50.111598707s) [noise=true , address=104.36.82.4:4803 , piece_id=7MSIWKFUIKVUANRYUKBRFNLVS6L7ZVFYMT6WB4TWBRPZEMOW34SQ, seg_pos={0 0}, upload=356247]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: context deadline exceeded; context deadline exceeded
Jul 11 22:07:23.185464926 (444h55m50.111759617s) [noise=true , address=104.36.82.14:4223 , piece_id=HBBHH4GTE4P3JY5KR5L42OPSBZV7Q7G7M4L564D2TDQPEV7MLLWQ, seg_pos={0 0}, upload=356247]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:45784->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:23.185569967 (444h55m50.111864643s) [noise=true , address=104.36.82.15:4259 , piece_id=D26DTTLWDTLRFFLVA3JU7D5NDNBJQLPT4FEDCQNSLHRYO47GZNBQ, seg_pos={0 0}, upload=356247]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 159.223.143.39:35752->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:23.185590519 (444h55m50.111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 30.029097478s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.XX...................................................XX..XX] duration:
30.001965729s
FAIL (2025-07-11 22:06:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.00199127s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [...................................................X..XX.XXX] duration:
30.030308492s
FAIL (2025-07-11 22:06:48 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
ot 8)
Jul 11 22:06:18.051185338 (444h54m44.977480033s) [upload=356238]: Starting upload
Jul 11 22:06:18.051318121 (444h54m44.977612817s) [upload=356238]: Done starting upload
Jul 11 22:06:18.051487943 (444h54m44.977782632s) [upload=356238]: Uploading segments...
Jul 11 22:06:18.051923180 (444h54m44.978217876s) [upload=356238]: Got next segment. Inline: false
Jul 11 22:06:18.051992272 (444h54m44.978286964s) [upload=356238]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:06:18.052020128 (444h54m44.978314816s) [upload=356238]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:18.105700657 (444h54m45.031995345s) [seg_pos={0 0}, upload=356238]: Begin upload segment...
Jul 11 22:06:18.105763227 (444h54m45.032057916s) [seg_pos={0 0}, upload=356238]: Done begin upload segment.
Jul 11 22:06:18.105789758 (444h54m45.032084447s) [noise=true , address=77.90.31.10:4518 , piece_id=DISXHPA7KYC2KKYMHKFDOL3GNNRAETGOUIBUZ3GM3TCDEHTGZOEQ, seg_pos={0 0}, upload=356238]: Uploading piece...
Jul 11 22:06:18.106022684 (444h54m45.032317362s) [noise=true , address=77.90.31.13:4627 , piece_id=5OKHNAVPMUVLBRFBN4T6WV357I2EE4F4ID4HTK5FXS6LDONEIPVA, seg_pos={0 0}, upload=356238]: Uploading piece...
Jul 11 22:06:18.106427098 (444h54m45.032721776s) [noise=true , address=104.36.82.10:4078 , piece_id=ZWTGOOBE4Y2IG3I43SD4QIXP5IU2N7SLLTSV37GAZWRSKNKBP65A, seg_pos={0 0}, upload=356238]: Uploading piece...
Jul 11 22:06:18.107423986 (444h54m45.033718677s) [noise=true , address=45.91.116.12:4227 , piece_id=V5MUOE53LDI6ZI77XSPLATYUV5BPCG7GWBLO6L2ILNRACNXPDJRA, seg_pos={0 0}, upload=356238]: Uploading piece...
Jul 11 22:06:18.108113717 (444h54m45.034408398s) [noise=true , address=104.36.82.15:4259 , piece_id=JUS4XDWTU36Z4QD57NU5XEBM5WIFBKOYFOAEBE655TJL77WJ464Q, seg_pos={0 0}, upload=356238]: Uploading piece...
Jul 11 22:06:18.108801938 (444h54m45.03509662s) [noise=true , address=104.36.82.14:4222 , piece_id=VUWXRL74AE2SQHXH6PB2JARXMCQLXKET3NNGI5PRS24ZVYR4P4DA, seg_pos={0 0}, upload=356238]: Uploading piece...
Jul 11 22:06:18.109474825 (444h54m45.035769513s) [noise=true , address=77.90.31.15:4697 , piece_id=WEHCZSJVYPCWAYUIBAFUZKXUXDGVWBKQ6K57GSJH354FOAWRQE2Q, seg_pos={0 0}, upload=356238]: Uploading piece...
Jul 11 22:06:18.110085284 (444h54m45.03637997s) [noise=true , address=104.36.82.6:4873 , piece_id=JVCBPXQR73Q3IN2VLNSTR2NR6FNTPNPSYKS7ASNDK5YR4QTWKGHA, seg_pos={0 0}, upload=356238]: Uploading piece...
Jul 11 22:06:18.110666176 (444h54m45.036960858s) [noise=true , address=45.91.116.5:4298 , piece_id=AV4HIPXRHWFDRHKRFSOCGPHKGVDJJJGPTASFYJWHJKEKLNZH4HYA, seg_pos={0 0}, upload=356238]: Uploading piece...
Jul 11 22:06:18.111303913 (444h54m45.037598594s) [noise=true , address=77.90.31.16:4735 , piece_id=JGG6BP6YJHZZUYEMOLXN5UU3NXXSRI4JQZDOVIGSU47FATVCEQNQ, seg_pos={0 0}, upload=356238]: Uploading piece...
Jul 11 22:06:18.111863776 (444h54m45.038158467s) [noise=true , address=77.90.31.17:4771 , piece_id=ZXMQJXXTXVKY3LOUVMXAQDXXSOUBKN2A53FJF6DERHWIVD7EJS2A, seg_pos={0 0}, upload=356238]: Uploading piece...
Jul 11 22:06:18.112422111 (444h54m45.038716789s) [noise=true , address=45.91.116.14:4189 , piece_id=LONAOHD7PA7GKNDEIW4GJPTZSVYRYW2T4TM3H27JBZSN7RKDJHQA, seg_pos={0 0}, upload=356238]: Uploading piece...
Jul 11 22:06:18.113012204 (444h54m45.039306885s) [noise=true , address=45.91.116.9:4261 , piece_id=S5DRNWQIIHA6Q7ASAJQ2AHV6CDEIWUIR3BW7LR5E6VZOGF55NK6Q, seg_pos={0 0}, upload=356238]: Uploading piece...
Jul 11 22:06:18.115891525 (444h54m45.042186201s) [noise=true , address=104.36.82.12:4149 , piece_id=IRTZMAJECCGE6BDPSRYRWDQIERNO7RVKB2J6Z3STLN55A453HKXQ, seg_pos={0 0}, upload=356238]: Uploading piece...
Jul 11 22:06:18.116450165 (444h54m45.042744837s) [noise=true , address=104.36.82.9:4007 , piece_id=R4XKPFVEZIPSIBS4YEGI7IZY3A5F3JT4SKKOVNFVRNXDGLI3HLOQ, seg_pos={0 0}, upload=356238]: Uploading piece...
Jul 11 22:06:18.116908391 (444h54m45.043203063s) [noise=true , address=104.36.82.14:4223 , piece_id=4FQRRH7Q5PU2UDW7G2IRZOR3MVZRDARCXXDOY7KY3DSQ7QXACMLQ, seg_pos={0 0}, upload=356238]: Uploading piece...
Jul 11 22:06:18.188681218 (444h54m45.114975895s) [upload=356238]: Next returned nil segment
Jul 11 22:06:18.188697258 (444h54m45.114991937s) [upload=356238]: Waiting for error group managing segments...
Jul 11 22:06:18.198308565 (444h54m45.124603255s) [noise=true , address=45.91.116.12:4227 , piece_id=V5MUOE53LDI6ZI77XSPLATYUV5BPCG7GWBLO6L2ILNRACNXPDJRA, seg_pos={0 0}, upload=356238]: Done uploading piece. err: <nil>
Jul 11 22:06:18.243789324 (444h54m45.170084015s) [noise=true , address=45.91.116.9:4261 , piece_id=S5DRNWQIIHA6Q7ASAJQ2AHV6CDEIWUIR3BW7LR5E6VZOGF55NK6Q, seg_pos={0 0}, upload=356238]: Done uploading piece. err: <nil>
Jul 11 22:06:18.257171408 (444h54m45.183466088s) [noise=true , address=45.91.116.14:4189 , piece_id=LONAOHD7PA7GKNDEIW4GJPTZSVYRYW2T4TM3H27JBZSN7RKDJHQA, seg_pos={0 0}, upload=356238]: Done uploading piece. err: <nil>
Jul 11 22:06:18.296692639 (444h54m45.222987326s) [noise=true , address=45.91.116.5:4298 , piece_id=AV4HIPXRHWFDRHKRFSOCGPHKGVDJJJGPTASFYJWHJKEKLNZH4HYA, seg_pos={0 0}, upload=356238]: Done uploading piece. err: <nil>
Jul 11 22:06:18.432435928 (444h54m45.358730608s) [noise=true , address=77.90.31.10:4518 , piece_id=DISXHPA7KYC2KKYMHKFDOL3GNNRAETGOUIBUZ3GM3TCDEHTGZOEQ, seg_pos={0 0}, upload=356238]: Done uploading piece. err: <nil>
Jul 11 22:06:18.485254644 (444h54m45.411549343s) [noise=true , address=77.90.31.13:4627 , piece_id=5OKHNAVPMUVLBRFBN4T6WV357I2EE4F4ID4HTK5FXS6LDONEIPVA, seg_pos={0 0}, upload=356238]: Done uploading piece. err: <nil>
Jul 11 22:06:18.909944371 (444h54m45.836239044s) [noise=true , address=77.90.31.16:4735 , piece_id=JGG6BP6YJHZZUYEMOLXN5UU3NXXSRI4JQZDOVIGSU47FATVCEQNQ, seg_pos={0 0}, upload=356238]: Done uploading piece. err: <nil>
Jul 11 22:06:19.558796563 (444h54m46.485091238s) [noise=true , address=77.90.31.15:4697 , piece_id=WEHCZSJVYPCWAYUIBAFUZKXUXDGVWBKQ6K57GSJH354FOAWRQE2Q, seg_pos={0 0}, upload=356238]: Done uploading piece. err: <nil>
Jul 11 22:06:19.945915442 (444h54m46.872210115s) [noise=true , address=77.90.31.17:4771 , piece_id=ZXMQJXXTXVKY3LOUVMXAQDXXSOUBKN2A53FJF6DERHWIVD7EJS2A, seg_pos={0 0}, upload=356238]: Done uploading piece. err: <nil>
Jul 11 22:06:48.022035645 (444h55m14.948330332s) [noise=true , address=104.36.82.6:4873 , piece_id=JVCBPXQR73Q3IN2VLNSTR2NR6FNTPNPSYKS7ASNDK5YR4QTWKGHA, seg_pos={0 0}, upload=356238]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:48.022120894 (444h55m14.948415588s) [noise=true , address=104.36.82.12:4149 , piece_id=IRTZMAJECCGE6BDPSRYRWDQIERNO7RVKB2J6Z3STLN55A453HKXQ, seg_pos={0 0}, upload=356238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:48.022194382 (444h55m14.948489069s) [noise=true , address=104.36.82.9:4007 , piece_id=R4XKPFVEZIPSIBS4YEGI7IZY3A5F3JT4SKKOVNFVRNXDGLI3HLOQ, seg_pos={0 0}, upload=356238]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:47818->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:48.022306683 (444h55m14.948601369s) [noise=true , address=104.36.82.15:4259 , piece_id=JUS4XDWTU36Z4QD57NU5XEBM5WIFBKOYFOAEBE655TJL77WJ464Q, seg_pos={0 0}, upload=356238]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 159.223.143.39:52300->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:48.022314931 (444h55m14.948609621s) [noise=true , address=104.36.82.14:4223 , piece_id=4FQRRH7Q5PU2UDW7G2IRZOR3MVZRDARCXXDOY7KY3DSQ7QXACMLQ, seg_pos={0 0}, upload=356238]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:41088->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:48.022375083 (444h55m14.94866977s) [noise=true , address=104.36.82.10:4078 , piece_id=ZWTGOOBE4Y2IG3I43SD4QIXP5IU2N7SLLTSV37GAZWRSKNKBP65A, seg_pos={0 0}, upload=356238]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:39860->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:48.022478018 (444h55m14.948772698s) [noise=true , address=104.36.82.14:4222 , piece_id=VUWXRL74AE2SQHXH6PB2JARXMCQLXKET3NNGI5PRS24ZVYR4P4DA, seg_pos={0 0}, upload=356238]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:35382->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:48.022496568 (444h55m14.948791256s) [seg_pos={0 0}, upload=356238]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Jul 11 22:06:48.050077138 (444h55m14.976371826s) [upload=356238]: Done uploading segments...
Jul 11 22:06:48.050101701 (444h55m14.976396386s) [upload=356238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 30.030329267s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.029184034s
FAIL (2025-07-11 22:06:47 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Begin upload segment...
Jul 11 22:05:48.115011540 (444h54m15.041306216s) [seg_pos={0 0}, upload=356232]: Done begin upload segment.
Jul 11 22:05:48.115076977 (444h54m15.041371666s) [noise=true , address=104.36.82.18:4369 , piece_id=NUCSK7GBSDCHWQ5GDGMGSPD677FGLPAG2NU7ZKQVBOR6ANAQCKFA, seg_pos={0 0}, upload=356232]: Uploading piece...
Jul 11 22:05:48.115438577 (444h54m15.041733269s) [noise=true , address=79.127.219.38:4034 , piece_id=ALLHZPUBKX6UJKPRB2WMBTAYVWONJEDU4YJHNXT65HRBWANF4AKA, seg_pos={0 0}, upload=356232]: Uploading piece...
Jul 11 22:05:48.115816062 (444h54m15.042110761s) [noise=true , address=45.91.116.14:4211 , piece_id=W5CAC3SOO3POEU2BVSVBGFYEP3XZQJUXKM5DQWE2KXTP3MVEU6ZA, seg_pos={0 0}, upload=356232]: Uploading piece...
Jul 11 22:05:48.116382152 (444h54m15.042676841s) [noise=true , address=64.49.13.154:4925 , piece_id=GDEY55ZY2HESA33OHQYLQRY2IK2NS5LOPQTCXS52KYW5QGBWYGLA, seg_pos={0 0}, upload=356232]: Uploading piece...
Jul 11 22:05:48.116665236 (444h54m15.042959929s) [noise=true , address=77.90.31.15:4722 , piece_id=4NSFGP3JAPTC6PK5TQT6H3N6OWPKFJTSPTEXWAULIRQLZTBQJ5CQ, seg_pos={0 0}, upload=356232]: Uploading piece...
Jul 11 22:05:48.116899835 (444h54m15.043194521s) [noise=true , address=104.36.82.49:4011 , piece_id=6LB5ZILMX3B6CGTJFWEWGJ6GQY4LWVL6QEB5JUHP5Z324PFGPRQQ, seg_pos={0 0}, upload=356232]: Uploading piece...
Jul 11 22:05:48.117372000 (444h54m15.043666686s) [noise=true , address=77.90.31.16:4757 , piece_id=2CZYVVBJQDJUQOHSGM5J6EIZKTX53GC72MF5VKZRZO6Z5C57WIQQ, seg_pos={0 0}, upload=356232]: Uploading piece...
Jul 11 22:05:48.117821745 (444h54m15.044116432s) [noise=true , address=77.90.31.19:4876 , piece_id=NZJZB2CJ2GJ6QYUFJC4ER3T6S22BVCRJRTQUWVYCCE7DIJKUHFWQ, seg_pos={0 0}, upload=356232]: Uploading piece...
Jul 11 22:05:48.122181601 (444h54m15.048476291s) [noise=true , address=104.36.82.4:4833 , piece_id=G7P3PPUSOU3OPMLDA4OYGR2MG2FPUAGGDNIQBO2OIW4RYR6BIQ5A, seg_pos={0 0}, upload=356232]: Uploading piece...
Jul 11 22:05:48.123325150 (444h54m15.049619838s) [noise=true , address=77.90.31.11:4558 , piece_id=XQ73S6O4ZJPVDON3MTW3UEEK6K2DIENLDAVKKYAKIQRZP5OC7KTQ, seg_pos={0 0}, upload=356232]: Uploading piece...
Jul 11 22:05:48.124117490 (444h54m15.050412178s) [noise=true , address=77.90.31.18:4812 , piece_id=XRMAYVPPD45JLOBJ5PJ32N222TFQE4BTTGDLYM4MF7Q7PBO64DDQ, seg_pos={0 0}, upload=356232]: Uploading piece...
Jul 11 22:05:48.124759188 (444h54m15.051053875s) [noise=true , address=104.36.82.19:4422 , piece_id=F3DDCJINPDSJC3LSQ6GWH7UZVP3WMBMKQ7OZTSF6A4VTOEDNUGQQ, seg_pos={0 0}, upload=356232]: Uploading piece...
Jul 11 22:05:48.125434795 (444h54m15.051729492s) [noise=true , address=45.91.95.3:4004 , piece_id=25XCJO2ZYDNRPEWZ56TBG4WPNDKU6D7VB433BCZ7VYKQEQCF4LIA, seg_pos={0 0}, upload=356232]: Uploading piece...
Jul 11 22:05:48.126376895 (444h54m15.052671582s) [noise=true , address=79.127.205.233:4645 , piece_id=XCX5ACOYFQRM7YLLPFQK52HTZTJ4EJ4CPI44L4GNW2FOPNO6LBKA, seg_pos={0 0}, upload=356232]: Uploading piece...
Jul 11 22:05:48.127105813 (444h54m15.053400513s) [noise=true , address=45.91.95.4:4034 , piece_id=QARMAW3PFAL4AWESAJCZY5V5AQHQ4OKEK4LG2HTD5YKIS5JZG7MA, seg_pos={0 0}, upload=356232]: Uploading piece...
Jul 11 22:05:48.127854457 (444h54m15.054149149s) [noise=true , address=77.90.31.10:4549 , piece_id=EWLT62524SNFO2WFCGXKZREVVR7OPTR5JTWCDUKCA6D66S3ASF3Q, seg_pos={0 0}, upload=356232]: Uploading piece...
Jul 11 22:05:48.128593233 (444h54m15.054887929s) [noise=true , address=104.36.82.36:4624 , piece_id=AIL4TPROEY4ZFVDGL7VLHNF5NHJDWFGEAT2H5AUNMHNA5E2XOBAA, seg_pos={0 0}, upload=356232]: Uploading piece...
Jul 11 22:05:48.129202345 (444h54m15.055497033s) [noise=true , address=104.36.82.22:4529 , piece_id=RPJFVUSMDAWTDEN3NYYYDFMAAW2GRMTSLYWEXL5DIV64MSEOPZLA, seg_pos={0 0}, upload=356232]: Uploading piece...
Jul 11 22:05:48.230646251 (444h54m15.156940967s) [upload=356232]: Next returned nil segment
Jul 11 22:05:48.230736952 (444h54m15.157031825s) [upload=356232]: Waiting for error group managing segments...
Jul 11 22:05:48.428282286 (444h54m15.35457696s) [noise=true , address=77.90.31.16:4757 , piece_id=2CZYVVBJQDJUQOHSGM5J6EIZKTX53GC72MF5VKZRZO6Z5C57WIQQ, seg_pos={0 0}, upload=356232]: Done uploading piece. err: <nil>
Jul 11 22:05:48.467746987 (444h54m15.394041674s) [noise=true , address=77.90.31.10:4549 , piece_id=EWLT62524SNFO2WFCGXKZREVVR7OPTR5JTWCDUKCA6D66S3ASF3Q, seg_pos={0 0}, upload=356232]: Done uploading piece. err: <nil>
Jul 11 22:05:48.519523101 (444h54m15.445817774s) [noise=true , address=77.90.31.18:4812 , piece_id=XRMAYVPPD45JLOBJ5PJ32N222TFQE4BTTGDLYM4MF7Q7PBO64DDQ, seg_pos={0 0}, upload=356232]: Done uploading piece. err: <nil>
Jul 11 22:05:48.619411172 (444h54m15.54570585s) [noise=true , address=45.91.116.14:4211 , piece_id=W5CAC3SOO3POEU2BVSVBGFYEP3XZQJUXKM5DQWE2KXTP3MVEU6ZA, seg_pos={0 0}, upload=356232]: Done uploading piece. err: <nil>
Jul 11 22:05:48.683338977 (444h54m15.609633663s) [noise=true , address=77.90.31.11:4558 , piece_id=XQ73S6O4ZJPVDON3MTW3UEEK6K2DIENLDAVKKYAKIQRZP5OC7KTQ, seg_pos={0 0}, upload=356232]: Done uploading piece. err: <nil>
Jul 11 22:05:48.779365743 (444h54m15.705660441s) [noise=true , address=79.127.205.233:4645 , piece_id=XCX5ACOYFQRM7YLLPFQK52HTZTJ4EJ4CPI44L4GNW2FOPNO6LBKA, seg_pos={0 0}, upload=356232]: Done uploading piece. err: <nil>
Jul 11 22:05:48.812161844 (444h54m15.738456517s) [noise=true , address=64.49.13.154:4925 , piece_id=GDEY55ZY2HESA33OHQYLQRY2IK2NS5LOPQTCXS52KYW5QGBWYGLA, seg_pos={0 0}, upload=356232]: Done uploading piece. err: <nil>
Jul 11 22:05:49.094040717 (444h54m16.020335394s) [noise=true , address=77.90.31.19:4876 , piece_id=NZJZB2CJ2GJ6QYUFJC4ER3T6S22BVCRJRTQUWVYCCE7DIJKUHFWQ, seg_pos={0 0}, upload=356232]: Done uploading piece. err: <nil>
Jul 11 22:05:49.448182353 (444h54m16.374477029s) [noise=true , address=77.90.31.15:4722 , piece_id=4NSFGP3JAPTC6PK5TQT6H3N6OWPKFJTSPTEXWAULIRQLZTBQJ5CQ, seg_pos={0 0}, upload=356232]: Done uploading piece. err: <nil>
Jul 11 22:05:49.457437503 (444h54m16.383732197s) [noise=true , address=79.127.219.38:4034 , piece_id=ALLHZPUBKX6UJKPRB2WMBTAYVWONJEDU4YJHNXT65HRBWANF4AKA, seg_pos={0 0}, upload=356232]: Done uploading piece. err: <nil>
Jul 11 22:05:50.633971772 (444h54m17.560266459s) [noise=true , address=45.91.95.3:4004 , piece_id=25XCJO2ZYDNRPEWZ56TBG4WPNDKU6D7VB433BCZ7VYKQEQCF4LIA, seg_pos={0 0}, upload=356232]: Done uploading piece. err: <nil>
Jul 11 22:05:51.092276788 (444h54m18.018571476s) [noise=true , address=45.91.95.4:4034 , piece_id=QARMAW3PFAL4AWESAJCZY5V5AQHQ4OKEK4LG2HTD5YKIS5JZG7MA, seg_pos={0 0}, upload=356232]: Done uploading piece. err: <nil>
Jul 11 22:06:47.920282828 (444h55m14.846577505s) [noise=true , address=104.36.82.4:4833 , piece_id=G7P3PPUSOU3OPMLDA4OYGR2MG2FPUAGGDNIQBO2OIW4RYR6BIQ5A, seg_pos={0 0}, upload=356232]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: write tcp 159.223.143.39:42606->104.36.82.4:4833: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:47.920385027 (444h55m14.846679703s) [noise=true , address=104.36.82.19:4422 , piece_id=F3DDCJINPDSJC3LSQ6GWH7UZVP3WMBMKQ7OZTSF6A4VTOEDNUGQQ, seg_pos={0 0}, upload=356232]: Done uploading piece. err: ecclient: upload failed (node:1k7Bs5nYevGzsVWhp85cgDsbrpzbefborGwMAAnDsbeT9x7gio, address:104.36.82.19:4422): protocol: write tcp 159.223.143.39:49700->104.36.82.19:4422: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:47.920445144 (444h55m14.846739823s) [noise=true , address=104.36.82.36:4624 , piece_id=AIL4TPROEY4ZFVDGL7VLHNF5NHJDWFGEAT2H5AUNMHNA5E2XOBAA, seg_pos={0 0}, upload=356232]: Done uploading piece. err: ecclient: upload failed (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1, address:104.36.82.36:4624): protocol: write tcp 159.223.143.39:46550->104.36.82.36:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:47.920451283 (444h55m14.846745962s) [noise=true , address=104.36.82.22:4529 , piece_id=RPJFVUSMDAWTDEN3NYYYDFMAAW2GRMTSLYWEXL5DIV64MSEOPZLA, seg_pos={0 0}, upload=356232]: Done uploading piece. err: ecclient: upload failed (node:12bNgTA4CY9ShB6Mi3QpceTEqNCr8ohCMHNVvU6S1qJp2TdVP1i, address:104.36.82.22:4529): protocol: write tcp 159.223.143.39:41026->104.36.82.22:4529: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:47.920465471 (444h55m14.846760165s) [noise=true , address=104.36.82.18:4369 , piece_id=NUCSK7GBSDCHWQ5GDGMGSPD677FGLPAG2NU7ZKQVBOR6ANAQCKFA, seg_pos={0 0}, upload=356232]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: write tcp 159.223.143.39:46636->104.36.82.18:4369: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:47.920475369 (444h55m14.846770048s) [noise=true , address=104.36.82.49:4011 , piece_id=6LB5ZILMX3B6CGTJFWEWGJ6GQY4LWVL6QEB5JUHP5Z324PFGPRQQ, seg_pos={0 0}, upload=356232]: Done uploading piece. err: ecclient: upload failed (node:1DV1WUL3dGBjH23a6xtetmqmLE9fER1FQf53gmYFxZiP25ra8N, address:104.36.82.49:4011): protocol: write tcp 159.223.143.39:34374->104.36.82.49:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:47.920515763 (444h55m14.846810502s) [seg_pos={0 0}, upload=356232]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:06:47.946958856 (444h55m14.873253535s) [upload=356232]: Done uploading segments...
Jul 11 22:06:47.946992392 (444h55m14.873287078s) [upload=356232]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1m0.029211407s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
6.068881844s
FAIL (2025-07-11 22:06:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.068906824s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [....................................................X..XX.XX] duration:
30.026480172s
FAIL (2025-07-11 22:06:17 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
:47.980180220 (444h54m14.906474923s) [upload=356231]: Done starting upload
Jul 11 22:05:47.980393101 (444h54m14.906687774s) [upload=356231]: Uploading segments...
Jul 11 22:05:47.980635324 (444h54m14.906930014s) [upload=356231]: Got next segment. Inline: false
Jul 11 22:05:47.980686579 (444h54m14.906981506s) [upload=356231]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:05:47.980692694 (444h54m14.906987387s) [upload=356231]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:05:48.095949859 (444h54m15.022244546s) [seg_pos={0 0}, upload=356231]: Begin upload segment...
Jul 11 22:05:48.096000124 (444h54m15.022294799s) [seg_pos={0 0}, upload=356231]: Done begin upload segment.
Jul 11 22:05:48.096030781 (444h54m15.022325475s) [noise=true , address=104.36.82.16:4293 , piece_id=RTOSG5QCV6OZJWZMBXG7HUZOINW76E7WZRKWN5EGL2EZWZPC7SDA, seg_pos={0 0}, upload=356231]: Uploading piece...
Jul 11 22:05:48.096044963 (444h54m15.022339654s) [noise=true , address=104.36.82.11:4115 , piece_id=ZPG3RN4JT4HY5CH443EXINKZJHNNGPXVII2SPI3VNTKHVTU32NZA, seg_pos={0 0}, upload=356231]: Uploading piece...
Jul 11 22:05:48.096449410 (444h54m15.02274409s) [noise=true , address=45.91.116.7:4009 , piece_id=MTYPSS5U6HOV4EJBTPAXVPGOBPXVB2RVODPE2NFVLQODOPCUQXIA, seg_pos={0 0}, upload=356231]: Uploading piece...
Jul 11 22:05:48.096464428 (444h54m15.02275912s) [noise=true , address=45.91.116.11:4117 , piece_id=MMJUNGT3QXKLZB2M2IZZZ3Q4FVIXYPXR3M2ZAFDRCD77TX7EQDJA, seg_pos={0 0}, upload=356231]: Uploading piece...
Jul 11 22:05:48.096814022 (444h54m15.023108711s) [noise=true , address=45.91.116.4:4937 , piece_id=Y5WFTBHSDVIFKGANPTV2KO5RO6MIZUGSEDCBH7P4AFJEJFDM62TA, seg_pos={0 0}, upload=356231]: Uploading piece...
Jul 11 22:05:48.097265810 (444h54m15.023560503s) [noise=true , address=104.36.82.6:4875 , piece_id=6GQQYJJMHESXF3JGISRZOCKUIHTACANFQIPKXM3RPYFSBFUW7IDA, seg_pos={0 0}, upload=356231]: Uploading piece...
Jul 11 22:05:48.097631972 (444h54m15.023926668s) [noise=true , address=104.36.82.13:4185 , piece_id=KKEXDC47PRUPCSFWJGA7JOBVR3I4OESLYTCXE64EJCZVEM44BDKA, seg_pos={0 0}, upload=356231]: Uploading piece...
Jul 11 22:05:48.097830028 (444h54m15.024124708s) [noise=true , address=104.36.82.16:4295 , piece_id=PZX6SPC6HYVCYLUU54XFWIC3EDNF4TYJYHND2ICXAVBXNIGLQJZA, seg_pos={0 0}, upload=356231]: Uploading piece...
Jul 11 22:05:48.098088023 (444h54m15.024382721s) [noise=true , address=45.91.116.14:4191 , piece_id=OQB7QNIHXB2UFOAJN3EGWXSYJ45WCAMMY43CFD7I3PBHCUUCFBYQ, seg_pos={0 0}, upload=356231]: Uploading piece...
Jul 11 22:05:48.098617262 (444h54m15.024911958s) [noise=true , address=45.91.116.7:4011 , piece_id=EVQ6MQSGB6H3FMR6BELV67R6KZBSQU6JFKPL3PTS6U53CM5MVICQ, seg_pos={0 0}, upload=356231]: Uploading piece...
Jul 11 22:05:48.098726263 (444h54m15.025020953s) [noise=true , address=45.91.116.9:4262 , piece_id=2JUUNBSCZVTZ5VRYQJ4VESXWC2UDG2MVUHSNGHCFZX4BS4H35EZA, seg_pos={0 0}, upload=356231]: Uploading piece...
Jul 11 22:05:48.099110606 (444h54m15.025405294s) [noise=true , address=104.36.82.4:4802 , piece_id=H5I7YPSONOIQJ75D6LIEV3PLOWDUVZZWWJZNE6L2PAZFK3CUKXOA, seg_pos={0 0}, upload=356231]: Uploading piece...
Jul 11 22:05:48.099539974 (444h54m15.025834674s) [noise=true , address=104.36.82.6:4876 , piece_id=K7FQW6OY7XTJ5W7WBYWFQ5PEH7PXRHSX2WDAUX6FYXVXUAQ2Z7UQ, seg_pos={0 0}, upload=356231]: Uploading piece...
Jul 11 22:05:48.099907677 (444h54m15.026202367s) [noise=true , address=45.91.116.6:4975 , piece_id=NHRR54PTNJNCDVKAN4YIQFNLUEKTI2TBXUZ7YYRDVVBTXN44OXTA, seg_pos={0 0}, upload=356231]: Uploading piece...
Jul 11 22:05:48.100416519 (444h54m15.02671122s) [noise=true , address=104.36.82.9:4005 , piece_id=3HU7MZ4VFIF6CZL4GPGGRYH6A6E5Y4AL7DZ5BWWLA5JQCI4V2ZGA, seg_pos={0 0}, upload=356231]: Uploading piece...
Jul 11 22:05:48.100886397 (444h54m15.027181086s) [noise=true , address=45.91.116.14:4190 , piece_id=AZMQN2ECUS5T2AZSSJEDSALJACFQ3DLDZYOKWK46FBZANV7POQFA, seg_pos={0 0}, upload=356231]: Uploading piece...
Jul 11 22:05:48.198354936 (444h54m15.12464963s) [upload=356231]: Next returned nil segment
Jul 11 22:05:48.198375659 (444h54m15.124670355s) [upload=356231]: Waiting for error group managing segments...
Jul 11 22:05:48.222725896 (444h54m15.149020573s) [noise=true , address=45.91.116.9:4262 , piece_id=2JUUNBSCZVTZ5VRYQJ4VESXWC2UDG2MVUHSNGHCFZX4BS4H35EZA, seg_pos={0 0}, upload=356231]: Done uploading piece. err: <nil>
Jul 11 22:05:48.245624413 (444h54m15.17191909s) [noise=true , address=45.91.116.7:4009 , piece_id=MTYPSS5U6HOV4EJBTPAXVPGOBPXVB2RVODPE2NFVLQODOPCUQXIA, seg_pos={0 0}, upload=356231]: Done uploading piece. err: <nil>
Jul 11 22:05:48.318695000 (444h54m15.244989688s) [noise=true , address=45.91.116.6:4975 , piece_id=NHRR54PTNJNCDVKAN4YIQFNLUEKTI2TBXUZ7YYRDVVBTXN44OXTA, seg_pos={0 0}, upload=356231]: Done uploading piece. err: <nil>
Jul 11 22:05:48.440263577 (444h54m15.366558255s) [noise=true , address=45.91.116.14:4191 , piece_id=OQB7QNIHXB2UFOAJN3EGWXSYJ45WCAMMY43CFD7I3PBHCUUCFBYQ, seg_pos={0 0}, upload=356231]: Done uploading piece. err: <nil>
Jul 11 22:05:48.466203541 (444h54m15.392498229s) [noise=true , address=45.91.116.4:4937 , piece_id=Y5WFTBHSDVIFKGANPTV2KO5RO6MIZUGSEDCBH7P4AFJEJFDM62TA, seg_pos={0 0}, upload=356231]: Done uploading piece. err: <nil>
Jul 11 22:05:48.490670250 (444h54m15.416964942s) [noise=true , address=45.91.116.7:4011 , piece_id=EVQ6MQSGB6H3FMR6BELV67R6KZBSQU6JFKPL3PTS6U53CM5MVICQ, seg_pos={0 0}, upload=356231]: Done uploading piece. err: <nil>
Jul 11 22:05:49.278444229 (444h54m16.204738907s) [noise=true , address=45.91.116.14:4190 , piece_id=AZMQN2ECUS5T2AZSSJEDSALJACFQ3DLDZYOKWK46FBZANV7POQFA, seg_pos={0 0}, upload=356231]: Done uploading piece. err: <nil>
Jul 11 22:05:49.288781242 (444h54m16.215075918s) [noise=true , address=45.91.116.11:4117 , piece_id=MMJUNGT3QXKLZB2M2IZZZ3Q4FVIXYPXR3M2ZAFDRCD77TX7EQDJA, seg_pos={0 0}, upload=356231]: Done uploading piece. err: <nil>
Jul 11 22:06:17.927330369 (444h54m44.853625048s) [noise=true , address=104.36.82.16:4295 , piece_id=PZX6SPC6HYVCYLUU54XFWIC3EDNF4TYJYHND2ICXAVBXNIGLQJZA, seg_pos={0 0}, upload=356231]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:17.927396206 (444h54m44.853690882s) [noise=true , address=104.36.82.4:4802 , piece_id=H5I7YPSONOIQJ75D6LIEV3PLOWDUVZZWWJZNE6L2PAZFK3CUKXOA, seg_pos={0 0}, upload=356231]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:17.927453876 (444h54m44.853748548s) [noise=true , address=104.36.82.6:4875 , piece_id=6GQQYJJMHESXF3JGISRZOCKUIHTACANFQIPKXM3RPYFSBFUW7IDA, seg_pos={0 0}, upload=356231]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:17.927465859 (444h54m44.853760535s) [noise=true , address=104.36.82.9:4005 , piece_id=3HU7MZ4VFIF6CZL4GPGGRYH6A6E5Y4AL7DZ5BWWLA5JQCI4V2ZGA, seg_pos={0 0}, upload=356231]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 159.223.143.39:41650->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:17.927517233 (444h54m44.853811906s) [noise=true , address=104.36.82.13:4185 , piece_id=KKEXDC47PRUPCSFWJGA7JOBVR3I4OESLYTCXE64EJCZVEM44BDKA, seg_pos={0 0}, upload=356231]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:17.927540032 (444h54m44.853834705s) [noise=true , address=104.36.82.11:4115 , piece_id=ZPG3RN4JT4HY5CH443EXINKZJHNNGPXVII2SPI3VNTKHVTU32NZA, seg_pos={0 0}, upload=356231]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:17.927590162 (444h54m44.853884837s) [noise=true , address=104.36.82.6:4876 , piece_id=K7FQW6OY7XTJ5W7WBYWFQ5PEH7PXRHSX2WDAUX6FYXVXUAQ2Z7UQ, seg_pos={0 0}, upload=356231]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:17.927601905 (444h54m44.853896577s) [noise=true , address=104.36.82.16:4293 , piece_id=RTOSG5QCV6OZJWZMBXG7HUZOINW76E7WZRKWN5EGL2EZWZPC7SDA, seg_pos={0 0}, upload=356231]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 159.223.143.39:45398->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:17.927621541 (444h54m44.853916231s) [seg_pos={0 0}, upload=356231]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Jul 11 22:06:17.951654921 (444h54m44.877949597s) [upload=356231]: Done uploading segments...
Jul 11 22:06:17.951678608 (444h54m44.877973297s) [upload=356231]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 30.02650542s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..XX...................................................XX..X] duration:
30.000612728s
FAIL (2025-07-11 22:06:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000650728s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....................................................X..XX.X] duration:
30.02912989s
FAIL (2025-07-11 22:05:32 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
) [upload=356221]: Done starting upload
Jul 11 22:05:02.884159228 (444h53m29.810453906s) [upload=356221]: Uploading segments...
Jul 11 22:05:02.884646718 (444h53m29.810941412s) [upload=356221]: Got next segment. Inline: false
Jul 11 22:05:02.884735270 (444h53m29.81102996s) [upload=356221]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:05:02.884741587 (444h53m29.811036278s) [upload=356221]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:05:03.239977502 (444h53m30.16627219s) [seg_pos={0 0}, upload=356221]: Begin upload segment...
Jul 11 22:05:03.240030256 (444h53m30.166324948s) [seg_pos={0 0}, upload=356221]: Done begin upload segment.
Jul 11 22:05:03.240067627 (444h53m30.166362321s) [noise=true , address=77.90.31.14:4663 , piece_id=MWFG3SRAEQAGEYMHC2B4URELR4HKO5VVQFM24JIQN7Z67KRJGKXA, seg_pos={0 0}, upload=356221]: Uploading piece...
Jul 11 22:05:03.241291554 (444h53m30.167586254s) [noise=true , address=104.36.82.12:4150 , piece_id=FGAQS3RUOB2MIEWIYJBRMUVRIW6APIMV3OCUEUU3MUQXQS53IVWA, seg_pos={0 0}, upload=356221]: Uploading piece...
Jul 11 22:05:03.242004370 (444h53m30.16829907s) [noise=true , address=77.90.31.15:4697 , piece_id=H7PP2ZHUDMBA56U24LQ2MVXGJOI2WAX3GHHI23DZ4TA2WLQQA44A, seg_pos={0 0}, upload=356221]: Uploading piece...
Jul 11 22:05:03.242634105 (444h53m30.168928796s) [noise=true , address=104.36.82.8:4972 , piece_id=UQRVUK463CRITTYFSTW2O4UJCCEKWSEP5S47XFLCWMT6RNVQ7YAA, seg_pos={0 0}, upload=356221]: Uploading piece...
Jul 11 22:05:03.245326330 (444h53m30.171621028s) [noise=true , address=104.36.82.4:4801 , piece_id=BJCGIMRIWSGBIEYOWHSMF2IPVYBMBES5S4XKSK4DLYGCQSTSXDJA, seg_pos={0 0}, upload=356221]: Uploading piece...
Jul 11 22:05:03.246525149 (444h53m30.172819842s) [noise=true , address=104.36.82.6:4875 , piece_id=C7B3BDSGAJ2GHBDZVNS5TT7CG5TLN3HC5YU6D34ZXH4IT5ROFSHQ, seg_pos={0 0}, upload=356221]: Uploading piece...
Jul 11 22:05:03.247487899 (444h53m30.1737826s) [noise=true , address=45.91.116.14:4191 , piece_id=OEO4DXNMXPJ5PA76BGZ6Y3LCBFWGLH5BMXE7B4YK45ABX5IQIK7A, seg_pos={0 0}, upload=356221]: Uploading piece...
Jul 11 22:05:03.251548359 (444h53m30.177843059s) [noise=true , address=104.36.82.9:4006 , piece_id=U32GM3PACPWQSLRV2XB3QOAD7QNZO2CAJJNNTGR2RKXCPUM33AOQ, seg_pos={0 0}, upload=356221]: Uploading piece...
Jul 11 22:05:03.252162851 (444h53m30.178457548s) [noise=true , address=77.90.31.17:4770 , piece_id=GTWSZVRL3N2VG4V4HS3CWR4U267CX6G5GEIRZEM5ZU4FSL7B4BIQ, seg_pos={0 0}, upload=356221]: Uploading piece...
Jul 11 22:05:03.252700926 (444h53m30.178995674s) [noise=true , address=104.36.82.11:4114 , piece_id=WS53CGGDRW3HNRJSBBL5YK2B7E2WQ6R6BDAWMLHKFWXPK72VAQFA, seg_pos={0 0}, upload=356221]: Uploading piece...
Jul 11 22:05:03.253313824 (444h53m30.179608524s) [noise=true , address=104.36.82.10:4078 , piece_id=3CJ65VUA6KA3NOXLBCSW74O25GLJIAQOUXEVO3XYNHFEAVBRNBZQ, seg_pos={0 0}, upload=356221]: Uploading piece...
Jul 11 22:05:03.253871152 (444h53m30.180165852s) [noise=true , address=45.91.116.7:4011 , piece_id=NSKYX7IY4Q3NNGMBLKPCHRO6L5RHEGX5DL2UCP5WKILQMDO47V7Q, seg_pos={0 0}, upload=356221]: Uploading piece...
Jul 11 22:05:03.254394701 (444h53m30.180689401s) [noise=true , address=104.36.82.15:4259 , piece_id=JXXW5UAD6LUU2Y2T3PYS66Q7U67B7KLPDIBJCQRADJSOQ6ZJDSTQ, seg_pos={0 0}, upload=356221]: Uploading piece...
Jul 11 22:05:03.254917385 (444h53m30.181212096s) [noise=true , address=45.91.116.13:4153 , piece_id=BOH3CMMW6PEGIJJUESWKLVA3PB4X4TTUMQXSEQVQXPGBIIF3HW4Q, seg_pos={0 0}, upload=356221]: Uploading piece...
Jul 11 22:05:03.255493675 (444h53m30.181788377s) [noise=true , address=45.91.116.12:4227 , piece_id=5JW2BIAFBXCKADZILAZEWDPKFLSHJBSBSPLT3LUJ3RCQ6FFNJYSQ, seg_pos={0 0}, upload=356221]: Uploading piece...
Jul 11 22:05:03.256013486 (444h53m30.182308169s) [noise=true , address=104.36.82.16:4295 , piece_id=NXON7BEX4QGXQWD2UFWX2A72P3IZGVHLCP3MEWLEZQ3IAITQQW2Q, seg_pos={0 0}, upload=356221]: Uploading piece...
Jul 11 22:05:03.484313136 (444h53m30.410607819s) [upload=356221]: Next returned nil segment
Jul 11 22:05:03.484333264 (444h53m30.410627942s) [upload=356221]: Waiting for error group managing segments...
Jul 11 22:05:03.560236060 (444h53m30.486530752s) [noise=true , address=77.90.31.14:4663 , piece_id=MWFG3SRAEQAGEYMHC2B4URELR4HKO5VVQFM24JIQN7Z67KRJGKXA, seg_pos={0 0}, upload=356221]: Done uploading piece. err: <nil>
Jul 11 22:05:03.741941810 (444h53m30.668236489s) [noise=true , address=77.90.31.17:4770 , piece_id=GTWSZVRL3N2VG4V4HS3CWR4U267CX6G5GEIRZEM5ZU4FSL7B4BIQ, seg_pos={0 0}, upload=356221]: Done uploading piece. err: <nil>
Jul 11 22:05:03.742025695 (444h53m30.668320384s) [noise=true , address=77.90.31.15:4697 , piece_id=H7PP2ZHUDMBA56U24LQ2MVXGJOI2WAX3GHHI23DZ4TA2WLQQA44A, seg_pos={0 0}, upload=356221]: Done uploading piece. err: <nil>
Jul 11 22:05:03.884844190 (444h53m30.811138879s) [noise=true , address=45.91.116.13:4153 , piece_id=BOH3CMMW6PEGIJJUESWKLVA3PB4X4TTUMQXSEQVQXPGBIIF3HW4Q, seg_pos={0 0}, upload=356221]: Done uploading piece. err: <nil>
Jul 11 22:05:04.819858320 (444h53m31.746152994s) [noise=true , address=45.91.116.12:4227 , piece_id=5JW2BIAFBXCKADZILAZEWDPKFLSHJBSBSPLT3LUJ3RCQ6FFNJYSQ, seg_pos={0 0}, upload=356221]: Done uploading piece. err: <nil>
Jul 11 22:05:05.336715838 (444h53m32.263010517s) [noise=true , address=45.91.116.7:4011 , piece_id=NSKYX7IY4Q3NNGMBLKPCHRO6L5RHEGX5DL2UCP5WKILQMDO47V7Q, seg_pos={0 0}, upload=356221]: Done uploading piece. err: <nil>
Jul 11 22:05:06.695211707 (444h53m33.621506383s) [noise=true , address=45.91.116.14:4191 , piece_id=OEO4DXNMXPJ5PA76BGZ6Y3LCBFWGLH5BMXE7B4YK45ABX5IQIK7A, seg_pos={0 0}, upload=356221]: Done uploading piece. err: <nil>
Jul 11 22:05:32.832607967 (444h53m59.758902644s) [noise=true , address=104.36.82.9:4006 , piece_id=U32GM3PACPWQSLRV2XB3QOAD7QNZO2CAJJNNTGR2RKXCPUM33AOQ, seg_pos={0 0}, upload=356221]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:05:32.832674596 (444h53m59.758969276s) [noise=true , address=104.36.82.10:4078 , piece_id=3CJ65VUA6KA3NOXLBCSW74O25GLJIAQOUXEVO3XYNHFEAVBRNBZQ, seg_pos={0 0}, upload=356221]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:05:32.832726736 (444h53m59.759021413s) [noise=true , address=104.36.82.15:4259 , piece_id=JXXW5UAD6LUU2Y2T3PYS66Q7U67B7KLPDIBJCQRADJSOQ6ZJDSTQ, seg_pos={0 0}, upload=356221]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:05:32.832781191 (444h53m59.759075867s) [noise=true , address=104.36.82.16:4295 , piece_id=NXON7BEX4QGXQWD2UFWX2A72P3IZGVHLCP3MEWLEZQ3IAITQQW2Q, seg_pos={0 0}, upload=356221]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:05:32.832840352 (444h53m59.759135027s) [noise=true , address=104.36.82.4:4801 , piece_id=BJCGIMRIWSGBIEYOWHSMF2IPVYBMBES5S4XKSK4DLYGCQSTSXDJA, seg_pos={0 0}, upload=356221]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: context deadline exceeded; context deadline exceeded
Jul 11 22:05:32.832889295 (444h53m59.759183971s) [noise=true , address=104.36.82.11:4114 , piece_id=WS53CGGDRW3HNRJSBBL5YK2B7E2WQ6R6BDAWMLHKFWXPK72VAQFA, seg_pos={0 0}, upload=356221]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:05:32.832935105 (444h53m59.759229781s) [noise=true , address=104.36.82.12:4150 , piece_id=FGAQS3RUOB2MIEWIYJBRMUVRIW6APIMV3OCUEUU3MUQXQS53IVWA, seg_pos={0 0}, upload=356221]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:05:32.833099222 (444h53m59.759393899s) [noise=true , address=104.36.82.8:4972 , piece_id=UQRVUK463CRITTYFSTW2O4UJCCEKWSEP5S47XFLCWMT6RNVQ7YAA, seg_pos={0 0}, upload=356221]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:05:32.833255311 (444h53m59.759549987s) [noise=true , address=104.36.82.6:4875 , piece_id=C7B3BDSGAJ2GHBDZVNS5TT7CG5TLN3HC5YU6D34ZXH4IT5ROFSHQ, seg_pos={0 0}, upload=356221]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 11 22:05:32.833275442 (444h53m59.759570134s) [seg_pos={0 0}, upload=356221]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:05:32.858907804 (444h53m59.785202484s) [upload=356221]: Done uploading segments...
Jul 11 22:05:32.858932934 (444h53m59.785227609s) [upload=356221]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 30.029155863s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
5.781951903s
FAIL (2025-07-11 22:05:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.781993307s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
11.324681003s
FAIL (2025-07-11 22:05:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 11.324708082s
core
uplink
us-select-1
us1-placement-20
|