Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [......XX..XXXX..............................................] duration:
463.909039ms
FAIL (2025-07-11 22:42:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 463.94135ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...............XX...........................................] duration:
525.066228ms
FAIL (2025-07-11 22:41:52 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 525.096877ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
683.925591ms
FAIL (2025-07-11 22:41:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 683.953543ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
784.923733ms
FAIL (2025-07-11 22:41:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 784.954017ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.......XX..XXXX.............................................] duration:
540.290383ms
FAIL (2025-07-11 22:41:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 540.33077ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.X..XX.XXXXX................................................] duration:
487.630592ms
FAIL (2025-07-11 22:41:14 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 487.660882ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [................XX..........................................] duration:
748.005745ms
FAIL (2025-07-11 22:40:58 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 748.04971ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [..X..XX.XXXXX...............................................] duration:
466.359341ms
FAIL (2025-07-11 22:40:54 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 466.392915ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [........XX..XXXX............................................] duration:
525.60191ms
FAIL (2025-07-11 22:40:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 525.649084ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [...X..XX.XXXXX..............................................] duration:
496.89524ms
FAIL (2025-07-11 22:40:34 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 496.924374ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.................XX.........................................] duration:
730.378832ms
FAIL (2025-07-11 22:40:34 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 730.406151ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
635.426742ms
FAIL (2025-07-11 22:40:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 635.542668ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
847.675616ms
FAIL (2025-07-11 22:40:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 847.812472ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [....X..XX.XXXXX.............................................] duration:
935.745381ms
FAIL (2025-07-11 22:40:03 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 935.802237ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..................XX........................................] duration:
576.688311ms
FAIL (2025-07-11 22:39:39 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 576.714166ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.........XX..XXXX...........................................] duration:
612.072158ms
FAIL (2025-07-11 22:39:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 612.10726ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..........XX..XXXX..........................................] duration:
15.569337122s
FAIL (2025-07-11 22:39:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 15.569362712s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.....X..XX.XXXXX............................................] duration:
527.359152ms
FAIL (2025-07-11 22:39:20 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 527.393962ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
696.698099ms
FAIL (2025-07-11 22:38:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 696.728364ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [......X..XX.XXXXX...........................................] duration:
500.401797ms
FAIL (2025-07-11 22:38:47 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 500.428516ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...................XX.......................................] duration:
784.034937ms
FAIL (2025-07-11 22:38:39 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 784.063936ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [...........XX..XXXX.........................................] duration:
591.554735ms
FAIL (2025-07-11 22:38:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 591.587278ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [....................XX......................................] duration:
513.148422ms
FAIL (2025-07-11 22:38:21 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 513.180705ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [............XX..XXXX........................................] duration:
525.202643ms
FAIL (2025-07-11 22:38:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 525.240239ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
603.064094ms
FAIL (2025-07-11 22:37:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 603.089727ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.......X..XX.XXXXX..........................................] duration:
459.669016ms
FAIL (2025-07-11 22:37:40 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 459.705198ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration:
550.198767ms
FAIL (2025-07-11 22:37:32 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 550.228397ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.............XX..XXXX.......................................] duration:
501.318696ms
FAIL (2025-07-11 22:37:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 501.363941ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
644.063519ms
FAIL (2025-07-11 22:37:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 644.114057ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
672.805455ms
FAIL (2025-07-11 22:36:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 672.835311ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [........X..XX.XXXXX.........................................] duration:
460.416881ms
FAIL (2025-07-11 22:36:44 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 460.450425ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [......................XX....................................] duration:
735.510428ms
FAIL (2025-07-11 22:36:43 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 735.541641ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..............XX..XXXX......................................] duration:
489.349219ms
FAIL (2025-07-11 22:36:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 489.382479ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.......................XX...................................] duration:
804.177776ms
FAIL (2025-07-11 22:36:25 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 804.207096ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
782.27183ms
FAIL (2025-07-11 22:36:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 782.295842ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.........X..XX.XXXXX........................................] duration:
557.715592ms
FAIL (2025-07-11 22:36:08 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 557.749101ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [...............XX..XXXX.....................................] duration:
516.960148ms
FAIL (2025-07-11 22:35:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 516.987034ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [..........X..XX.XXXXX.......................................] duration:
448.347616ms
FAIL (2025-07-11 22:35:42 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 448.381543ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [........................XX..................................] duration:
715.607282ms
FAIL (2025-07-11 22:35:39 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 715.638828ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
726.020486ms
FAIL (2025-07-11 22:35:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 726.042973ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [...........X..XX.XXXXX......................................] duration:
461.778813ms
FAIL (2025-07-11 22:35:22 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 461.805544ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.........................XX.................................] duration:
799.295026ms
FAIL (2025-07-11 22:35:14 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 799.348283ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [................XX..XXXX....................................] duration:
553.141689ms
FAIL (2025-07-11 22:35:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 553.163849ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
803.338631ms
FAIL (2025-07-11 22:35:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 803.368432ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [............X..XX.XXXXX.....................................] duration:
582.229744ms
FAIL (2025-07-11 22:35:01 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 582.328805ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.................XX..XXXX...................................] duration:
625.930303ms
FAIL (2025-07-11 22:34:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 625.96105ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.............X..XX.XXXXX....................................] duration:
492.106976ms
FAIL (2025-07-11 22:34:41 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 492.141014ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
707.764887ms
FAIL (2025-07-11 22:34:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 708.008838ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..........................XX................................] duration:
808.911015ms
FAIL (2025-07-11 22:34:09 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 808.942857ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..................XX..XXXX..................................] duration:
697.047858ms
FAIL (2025-07-11 22:34:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 697.091729ms
edge
gateway
us-select-1
|